About the job
Join us as an SDET - Test Automation AI (Agentic & LLM Systems) at Resillion!
As an integral part of our team, you will play a crucial role in defining and implementing assurance strategies for AI-enabled systems. Your focus will be on ensuring that our AI solutions are reliable, robust, explainable, secure, and fit for purpose throughout their entire lifecycle.
The AI Assurance Engineer will be responsible for assuring both:
- Probabilistic components (data, models, and AI outputs)
- Deterministic components (software, integrations, and infrastructure)
This role requires a comprehensive understanding of how to assure AI systems holistically rather than specializing in a single domain.
Key Responsibilities:
- AI Assurance Strategy:
- Define and implement an AI assurance strategy aligned with business risk and regulatory requirements.
- Ensure assurance coverage across the entire AI system lifecycle (design, build, deploy, operate).
- Collaborate with engineering, data, and product teams to integrate quality and risk controls from the outset.
- Probabilistic Component Assurance:
- Design validation methodologies for:
- Data quality and bias
- Model and prompt behavior
- Output accuracy, relevance, and consistency
- Implement evaluation techniques for:
- Drift and instability
- Hallucination and error patterns
- Facilitate human-in-the-loop reviews as necessary.
- Design validation methodologies for:
- Deterministic Component Assurance:
- Ensure the integrity of non-AI system elements including:
- Application logic and workflows
- APIs and integrations
- Security and access controls
- Design and execute:
- Functional testing
- Non-functional testing (performance, resilience, scalability)
- Security and data protection validation
- Ensure the integrity of non-AI system elements including:
- Automation & E2E Assurance:
- Develop automated assurance solutions for AI systems.
- Integrate assurance processes into CI/CD and deployment pipelines.
- Establish regression and quality gates across data, models, and orchestration workflows.
- Maintain a comprehensive assurance pipeline from input data to system outputs.
- Operational AI & Observability:
- Support monitoring and observability for AI-enabled systems in production.
- Analyze operational signals such as:
- Latency and failures
- Behavior changes
- Performance degradation