Senior Data Engineer
Skills
AdoptionApache SparkContinuous ImprovementData AnalyticsDriftEvolutionModeling
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- 5–7+ years of hands-on data engineering experience
- Advanced proficiency in Python SQL PySpark and ETL/ELT frameworks
- Strong experience with Databricks and distributed data processing
- Demonstrated ability to optimize SQL and Spark workloads for performance and cost
- Experience using AI development tools such as GitHub Copilot Databricks Assistant and SQL/IDE AI copilots
- Solid understanding of data modeling governance and CI/CD for data pipelines
- Excellent communication skills and ability to work in a cross-functional environment
- Experience working in Agile environments with Jira and Confluence
Nice to have
- Passionate
- Motivated
- Resourceful
- Innovative
- Forward thinking
- Committed
Day to day
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL
- Optimize SQL and Spark workloads for performance, cost efficiency, and reliability while supporting modern data platform initiatives
- Build robust ingestion frameworks, transformation logic, and reusable data engineering components for enterprise data assets
- Use AI coding copilots and automation tools to accelerate development, improve documentation, and enhance pipeline reliability
- Collaborate cross-functionally and mentor junior engineers to deliver high-quality data solutions in an Agile environment