LLMOps Engineer

Компания "Andersen"

Andersen is hiring an LLMOps Engineer for a project developing scalable AI systems, intelligent workflows, and production-ready LLM-powered solutions.

The customer is an international company delivering professional and technology-enabled solutions that support effective collaboration, structured communication, and operational efficiency for organizations. It operates in a fast-growing environment, focusing on scalability, security, and continuous improvement while developing digital platforms used by diverse clients worldwide.

The project is focused on developing and scaling an LLMOps platform that supports AI-powered capabilities such as semantic search, retrieval pipelines, and intelligent workflows. It includes building evaluation, monitoring, and quality frameworks to ensure reliable, scalable, and production-ready AI systems.

Responsibilities:

• Owning and evolving the LLMOps platform, including architecture decisions, integrations, access management, and environment maintenance.
• Defining and enforcing instrumentation standards for tracing LLM interactions, RAG pipelines, and agentic workflows.
• Partnering with product managers and engineering teams for achieving and maintaining instrumentation coverage across AI capabilities.
• Integrating observability capabilities into CI/CD processes for surfacing quality signals before production releases.
• Evaluating, adopting, and maintaining LLMOps tooling to ensure long-term platform effectiveness.
• Defining AI evaluation strategies together with product stakeholders, including metrics, scoring approaches, and quality thresholds.
• Establishing evaluation adequacy criteria for different AI features and risk levels, proactively identifying coverage gaps.
• Designing, building, and maintaining versioned golden datasets covering real-world scenarios and edge cases.
• Owning evaluation pipelines, including LLM-as-judge methodologies, heuristic evaluations, and human feedback mechanisms.
• Establishing prompt regression testing processes and A/B experimentation frameworks.
• Delivering AI quality reporting with actionable insights for engineering and product teams.
• Owning observability dashboards and alerting systems for latency, cost, quality metrics, and production AI failure signals.
• Defining and implementing AI cost optimization measures, including budgeting, caching, rate limiting, and usage visibility.
• Monitoring guardrails and content safety metrics as part of AI observability practices.
• Proactively identifying degradation trends, provider changes, model shifts, and production anomalies.
• Maintaining operational runbooks for common LLM failure scenarios and leading incident response activities related to AI quality.
• Defining reusable observability and evaluation standards for scalable adoption across teams.
• Building and maintaining shared libraries for simplifying instrumentation implementation.
• Ensuring compliance with SOC 2, ISO 27001, PII handling requirements, and data residency regulations.
• Translating evaluation outcomes and quality insights into actionable recommendations for engineering, product, and leadership stakeholders.
• Scaling observability and evaluation practices across additional product areas through platform capabilities and standardized processes.

Must-have:

• Experience owning AI systems, ML infrastructure, or software engineering for 6+ years.
• Bachelor’s degree in Computer Science, Engineering, IT, or related field.
• Experience building and scaling LLMOps / AI observability platforms in production.
• Experience designing AI evaluation frameworks (golden datasets, scoring, regression testing, LLM-as-judge).
• Hands-on experience with AI observability tools (Arize, Langfuse, W&B, LangSmith, MLflow).
• Strong observability knowledge (distributed tracing, logging, metrics, alerting, OpenTelemetry, Azure Monitor).
• Proficiency in Python and working knowledge of C#/.NET.
• Understanding of RAG, vector search, prompt engineering, and orchestration frameworks (LangChain, Semantic Kernel, LlamaIndex).
• Experience with CI/CD, Git workflows, and Azure DevOps.
• Experience managing LLM costs (tokens, caching, budget controls).
• Experience in regulated environments (SOC 2, ISO 27001).
• Experience with AI-assisted development tools.
• Level of English – Intermediate+ and above.

Nice to have:

• Experience with AI evaluation frameworks (DeepEval, LangTest).
• Familiarity with LLM tracing and OpenTelemetry integrations.
• Experience monitoring AI safety and guardrail systems.
• Experience with A/B testing for AI features.
• Understanding of embeddings, semantic similarity, and vector databases.
• Background in QA, ML Engineering, or Data Engineering.
• Experience scaling AI quality and observability practices.

Reasons why this job would be interesting to you:

• Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
• The opportunity to change the project and/or develop expertise in an interesting business domain.
• Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
• Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
• The opportunity to earn an additional up to 1,000 USD per month by participating in the company's activities.
• Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
• Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
• Certification compensation (AWS, PMP, etc).
• Referral program.
• Private health insurance and compensation for sports activities.

Join us!

Your personal data is protected in accordance with GDPR regulations.