LastWeek

LastThreeDays

Today

Aoristou

FullTime

Hybrid

As Senior Manager, DevOps Engineer (Individual Contributor), you will be the build, release, and reliability backbone for our Python SDK—a pip-installable package that provides reusable, modular AI capabilities (RAG components, agent/tool cookiecutters, MCP adaptors, standardized interfaces). You will design automated pipelines that enforce code quality, security, reproducibility, benchmarking, and integration tests for the SDK and ensure compatibility and operability for platform services that consume it.You’ll collaborate closely with:<ul><li>Embedded AI Architecture &amp; Solution Design to prioritize capabilities and validate interfaces against real solution patterns.</li><li>AI Engineering to align PI increments and integrate external technologies into the SDK.</li><li>Digital Creation Centers &amp; Forward Impact Teams (FITs) as consumers of the SDK in products/services, capturing feedback to refine release ergonomics.</li><li>Evaluation &amp; QA, Developer Experience, and LLMOps &amp; AI Integration engineers to make quality and operability automatic.</li></ul>Why this Role MattersFor a Python SDK to scale safely, we need rock‑solid packaging, dependency hygiene, and CI/CD: reproducible builds, secure artifacts, versioning discipline, and automated tests/benchmarks that guard against regressions. Your work turns modular capabilities into enterprise‑grade libraries that downstream services can rely on—ensuring consistent behavior, faster time‑to‑first‑success, and audit‑ready releases.ROLE RESPONSIBILITIES1) Python Packaging, Build &amp; Distribution<ul><li>Own build and release for the SDK: wheel/sdist creation, and reproducible builds.</li><li>Enforce semantic versioning, backward compatibility checks, and changelogs; design safe migration paths (deprecation policies, compatibility shims).</li><li>Standardize artifact repositories: internal PyPI (Dev/Stage/Prod indices), package signing, and promotion workflows with automated quality gates.</li></ul>2) CI/CD for Libraries &amp; Supply Chain Security<ul><li>Design CI/CD pipelines (e.g., GitHub Actions/Azure DevOps/GitLab CI) to automate linting, type checking, unit/integration tests, coverage, and style enforcement (black/ruff/mypy/pytest).</li><li>Integrate dependency auditing and vulnerability scanning, license compliance, and SBOM generation; implement artifact signing and provenance (SLSA‑like controls).</li><li>Codify policy‑as‑code (e.g., OPA/Conftest) so releases only ship when tests, coverage, docs, and security gates pass.</li></ul>3) Benchmarking, Performance &amp; Integration Testing<ul><li>Collaborate with Evaluation &amp; QA Engineers to build benchmark harnesses for RAG components, connectors, rankers, and agent/tool modules; track latency/cost envelopes and regressions over time.</li><li>Orchestrate integration tests against common stacks (vector databases, storages, identity providers) to validate interface contracts used by platform services.</li><li>Maintain regression baselines and performance dashboards; fail builds on significant regressions.</li></ul>4) Operability for Consumers (SDK↔Service Parity)<ul><li>Embed observability hooks appropriate for libraries (structured logging, configurable metrics/traces) so services consuming the SDK can integrate telemetry consistently.</li><li>Define interface contracts and runtime configuration patterns (config schema, secrets management integration points) that downstream services must honor.</li><li>Partner with platform teams to publish service readiness guides built on SDK interfaces (env parity, retry/backoff, error models).</li></ul>5) Environments, Reproducibility &amp; Tooling<ul><li>Design isolated test environments and fixtures for data, vector stores, and external dependencies; leverage tox/pytest‑xdist for matrix runs.</li><li>Own Infrastructure-as-Code limited to test/development environments and CI runners; ensure deterministic builds and environment parity.</li><li>Optimize pipeline performance (caching wheels, parallelization) and track DORA‑style delivery metrics to continuously improve.</li></ul>6) Developer Experience &amp; Contribution Workflows<ul><li>Collaborate with Developer Experience Engineers to produce developer‑ready artifacts (quickstarts, sample apps, docs bundles generated during CI).</li><li>Standardize contribution workflows: feature branches, PR checks, mandatory tests/benchmarks/docs acceptance criteria; enable community extensions safely.</li><li>Provide CLI/tooling or cookiecutters that scaffold new SDK modules with tests, docs, and benchmark hooks prewired.</li></ul>7) Collaboration &amp; Continuous Improvement<ul><li>Align pipeline policies with AI Capability Architecture Lead and ensure SDK interface consistency for downstream service teams.</li><li>Close feedback loops with Embedded Architecture, Solution Design, AI Engineering, Creation Centers, and FITs to prioritize pipeline and release ergonomics improvements.</li><li>Lead post‑release retrospectives to reduce defects, MTTR, and onboarding friction for consuming teams.</li></ul>QUALIFICATIONSBasic Qualifications<ul><li>7+ years in DevOps/SRE/Build &amp; Release engineering, with significant experience in Python packaging and library delivery (wheels/sdists, internal PyPI).</li><li>Expert in CI/CD for libraries (GitHub Actions/Azure DevOps/GitLab CI) and artifact management (PyPI/private indices, container registries as needed for test harnesses).</li><li>Strong hands‑on with cloud‑native development and CI runners; proficiency with IaC for build/test environments.</li><li>Proven track record implementing quality gates: automated tests, secrets management, test coverage, dependency/security scanning, SBOMs/signing, policy‑as‑code.</li><li>Experience embedding observability hooks for libraries and defining interface contracts for service consumers.</li></ul>Preferred Qualifications<ul><li>Familiarity with GenAI/LLM ecosystems (RAG pipelines, vector stores, agent/tool frameworks, MCP adaptors) and their benchmarking/testing needs.</li><li>Background in performance engineering for Python (profiling, async/concurrency, I/O, data pipelines) and managing latency/cost regressions.</li><li>Experience with open-source contribution workflows, multi‑language SDK delivery (Python primary; TypeScript/Java nice-to-have), and semantic versioning discipline.</li><li>Exposure to regulated environments and audit‑ready release documentation.</li><li>Comfortable partnering across architecture, QA, DX, LLMOps, platform, and product teams in a matrixed organization.</li></ul>

Senior Manager, DevOps 4949161

As Senior Manager, LLMOps &amp; AI Integration Engineer (Individual Contributor), you will lead the model integration, agent/tooling, and LLM operations strategy for our pip‑installable AI Capabilities SDK. You will design and maintain clean, extensible interfaces for LLMs, embeddings, retrieval, tool execution, and agentic workflows; build runtime guards and evaluation hooks; and optimize latency, reliability, and cost—so teams can embed AI safely and predictably.You will collaborate closely with:<ul><li>Embedded AI Architecture and Solution Design to prioritize capabilities and define interfaces aligned to real-world solution patterns.</li><li>AI Engineering to deliver PI increments, integrate external technologies, and codify model/agent failure modes and mitigations.</li><li>Digital Creation Centers and Forward Impact Teams (FITs) as consumers of the SDK, capturing feedback to refine ergonomics and performance.</li><li>DevOps, Evaluation &amp; QA, and Developer Experience engineers to wire guardrails, evaluation, and developer-ready assets into CI/CD and docs.</li></ul>Why this Role MattersLLMOps sits at the intersection of capability, safety, and operability. By turning complex model/agent behaviors into well-abstracted, evaluable, and observable SDK modules, you enable teams to deliver AI features fast with safety—reducing bespoke implementations, ensuring reproducibility, and controlling latency/cost at scale. Your work is foundational for platform services that rely on the SDK as their backbone.ROLE RESPONSIBILITIES1) Model &amp; Agent Integration Architecture<ul><li>Design Python SDK interfaces for LLM clients, embeddings, tokenization, and structured outputs (e.g., Pydantic/JSON schemas).</li><li>Implement function/tool calling abstractions, agent orchestration patterns (ReAct-like, planner/executor), and MCP adaptors for interoperable tools.</li><li>Provide configuration and secrets integration points that downstream services can adopt consistently (env vars, config schemas, key management hooks).</li></ul>2) Retrieval &amp; RAG Components<ul><li>Build RAG modules (document loaders, chunking/segmentation, embeddings, retrievers, rerankers) with pluggable backends (vector stores, search indices).</li><li>Standardize connectors and interface contracts for common data sources; ensure reproducible pipelines and SDK↔service parity of behaviors.</li><li>Optimize retrieval quality via evaluation hooks (precision/recall, MRR, hit rate) and tunable parameters (top‑k, thresholds).</li></ul>3) Guardrails, Safety &amp; Policy Integration<ul><li>Codify safety guardrails: prompt hygiene, jailbreak resistance, content filters, sensitive topic/policy checks, red‑team/adversarial probes.</li><li>Integrate privacy/security controls: data minimization, PII detection flags, safe logging, and deterministic truncation strategies.</li><li>Provide policy‑as‑code hooks and pre/post‑processing middleware to enforce runtime constraints within SDK flows.</li></ul>4) Performance, Reliability &amp; Cost Engineering<ul><li>Implement caching, batching, retry/backoff, circuit breakers, and fall‑back strategies across model/tool calls.</li><li>Profile and tune latency, throughput, and token usage/cost; expose configuration knobs and budget guards (token and cost caps).</li><li>Maintain benchmark suites and regression baselines for model and retrieval components; block releases on significant degradations.</li></ul>5) Observability‑by‑Design for Libraries<ul><li>Embed telemetry hooks (structured logs, metrics, traces) suitable for a library so that services consuming the SDK can attach enterprise observability.</li><li>Define SLIs relevant to LLMOps (latency, error rates, cache hit ratio, token usage, cost per call) and document runbook guidance for downstream teams.</li><li>Standardize error models and return types to ensure predictable handling across consuming services.</li></ul>6) Evaluation &amp; CI/CD Integration<ul><li>Partner with Evaluation &amp; QA Engineers to build test harnesses (unit/integration/contract/fuzz/adversarial) and benchmark suites for agents/tools, RAG, and model endpoints.</li><li>Work with DevOps Engineers to wire quality gates into CI/CD: coverage thresholds, performance budgets, safety checks, SBOM/signing for release artifacts.</li><li>Provide fixtures and simulators to enable deterministic tests across backends (vector stores, providers).</li></ul>7) Developer Experience &amp; Enablement<ul><li>Collaborate with Developer Experience and Technical Writers to deliver quickstarts, sample apps, notebooks, and code recipes for common patterns (RAG, agents, tool use, structured outputs).</li><li>Contribute to cookiecutters/CLI scaffolds that prewire configuration, tests, telemetry, and guardrails for new modules or integrations.</li><li>Support community contribution workflows (feature branches, PR checks, doc acceptance criteria) and mentor contributors on best practices.</li></ul>8) Collaboration &amp; Continuous Improvement<ul><li>Close feedback loops with Embedded Architecture, Solution Design, AI Engineering, Creation Centers, and FITs to refine APIs, defaults, and ergonomics.</li><li>Participate in post‑release retrospectives and incident reviews to strengthen guardrails, performance tuning, and developer experience.</li><li>Track and improve LLMOps KPIs (latency, cost per call, error rates, cache hit ratios, evaluation pass rates).</li></ul>MEASURES OF SUCCESS<ul><li>Performance &amp; cost: Latency and token/cost budgets consistently met; cache hit ratios improved; fewer performance regressions.</li><li>Safety &amp; reliability: Guardrail tests pass; reduced policy violations and incident rates; predictable error handling.</li><li>Adoption &amp; DX: Faster time‑to‑first‑success for consuming teams; positive developer feedback on APIs and examples.</li><li>Evaluation coverage: Benchmarks and adversarial tests integrated; high pass rates; clear baselines with trend reporting.</li><li>SDK↔Consumer parity: Consistent behavior across downstream services; reduced bespoke integrations and rework.</li></ul>QUALIFICATIONSBasic Qualifications<ul><li>7+ years in software/AI engineering with significant experience in LLMOps or model/tool integration for Python libraries.</li><li>Deep proficiency in Python and building extensible SDKs (interfaces, packaging, semantic versioning, backward compatibility).</li><li>Hands‑on with GenAI/LLM ecosystems (e.g., tool/function calling, structured outputs, prompt orchestration) and RAG components (embeddings, retrievers, rerankers, vector stores).</li><li>Strong grasp of performance engineering (profiling, caching, batching), resilience patterns (retry/backoff, circuit breakers), and cost control (token budgeting).</li><li>Experience embedding observability hooks and integrating evaluation/benchmarks into CI/CD pipelines.</li></ul>Preferred Qualifications<ul><li>Familiarity with agent frameworks, MCP adaptors, and multi‑provider model client abstractions.</li><li>Experience with adversarial/safety testing, privacy/PII considerations, and policy‑as‑code guardrails.</li><li>Background in open-source workflows (feature branches, PR reviews) and multi‑backend integration (vector DBs, search, storage).</li><li>Exposure to regulated environments (healthcare/pharma/finance) and audit‑ready evidence capture.</li><li>Comfortable collaborating across architecture, DevOps, QA, DX, platform, and product teams in matrixed organizations.</li></ul>

Θέσεις Εργασίας Φύλακας Μακεδονία, Πληροφορική

senior manager, devops 4949161

Pfizer Hellas

senior manager, llmops & ai integration engineer 4949157

Pfizer Hellas

Jobily.gr

Για Εργοδότες

Για Υποψηφίους

Εταιρεία