How Remoteville checks and expires listings
Senior AI Test Automation Engineer
Skills
Analytical SkillsData ScienceDriftEnd-To-End TestingFoundationsFundamentalsIncident Management
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- Strong grounding in test design exploratory testing automation frameworks debugging techniques and QA processes
- Strong experience with Python for automation and test harness creation
- Good experience in AI model validation and interpreting model outputs
- Ability to understand training pipelines overfitting drift hallucinations and accuracy metrics
- Skills in handling structured and unstructured data and basic statistical analysis
- Ability to identify bias and validate dataset quality
- Experience with API testing model checks and CI/CD integration
- Ability to work with governance and security standards
- Strong communication skills for collaboration and knowledge transfer
Nice to have
- Analytical
- Collaborative
- Detail-oriented
- Methodical
- Proactive
Day to day
- Develop an end-to-end testing strategy for AI services including test design automation frameworks and CI/CD integration
- Define implement and track AI evaluation metrics such as accuracy bias drift hallucination detection and LLM-as-judge
- Build and maintain a structured repository of test findings tool evaluations implementation details and lessons learned
- Design an operational support model with real-time monitoring incident management and iterative improvement for live AI services
- Collaborate with Data Science Architecture and Service Delivery teams to diagnose performance issues and transfer knowledge
Hiring process
- Please apply by submitting your latest CV for immediate review by our Talent team