How Remoteville checks and expires listings
AI Evaluation Engineer
Skills
Artificial IntelligenceCamerasDesignEngineeringHTMLLarge Language ModelsLiteracy
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- 3+ years of engineering experience
- At least 1 year focused on LLM/ML evaluation applied AI or data quality systems
- Hands-on experience with LLM evaluation frameworks
- Strong grasp of LLM-as-judge methodology and failure modes
- Statistical literacy and experimental design skills
- Product instincts for identifying key failure modes
- Strong written communication skills
Nice to have
- Detail-oriented
- Collaborative
- Analytical
- Independent
- Fast-moving
- Research-minded
Day to day
- Design and maintain evaluation infrastructure for LLM-powered features using tools like Promptfoo and LLM-as-judge methods.
- Translate fuzzy customer outcomes into precise rubrics and measurable quality criteria with product and domain experts.
- Instrument production traffic curate golden datasets and run rapid experiments to detect regressions and quantify tradeoffs across models prompts and agents.
- Evaluate multi-step agent behavior tool use correctness trajectory quality and end-to-end task completion across complex workflows.
- Create reusable templates documentation and tooling so engineers can treat evals as a normal part of development.
Hiring process
- Record a Loom video with camera and screen
- Walk through a project or code you're proud of
- Email the video to careers@firstignite.com within 48 hours
- Include the position title in the email subject line