Research Crawling Engineer
Skills
Chrome DevtoolsCuratingHypertext Transfer ProtocolLarge Language ModelsModelingProxy ServerPython
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- Extensive programming experience in Go Rust Python Java or C++
- Proven experience building web crawlers or large-scale data pipelines
- Solid understanding of HTTP networking protocols and browser behavior
- Familiarity with distributed systems and parallel processing techniques
- Experience handling large datasets at terabyte to petabyte scale
- Ability to debug and maintain systems in unstable or adversarial environments
Nice to have
- Technical
- High-output
- Low-ego
- Analytical
- Adaptable
Day to day
- Design and operate large-scale web crawling systems that collect high-quality public data at massive scale.
- Build high-throughput, fault-tolerant acquisition pipelines that handle millions to billions of URLs per day.
- Work through anti-bot defenses dynamic websites and complex networking challenges while improving crawl performance and data quality.
Hiring process
- Recruiter Coordination Call
- Hiring Manager Interview
- Founder / CEO Interview
- Secondary Executive Interview
- Final Interview