What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- 2+ years in a data role
- Dealing with large volumes of data (100+ GB/day)
- Python & Spark
- Event-driven data pipelines in AWS
- Kafka streaming applications
- RDBMS and NoSQL data stores
- SQL and query optimisation
- Testing serverless applications in a cloud environment
- Infrastructure-as-code and configuration-as-code (preferably Terraform)
- Source code management (Git)
- Continuous integration and deployment techniques
Day to day
- Joining our Data Engineering function, providing solutions for data management, organisation, and analysis.
- Working on a cloud-based platform, implementing back office search & reporting and real-time data streaming.
- Creating ETL pipelines to process and transform large volumes of data.
- Developing custom regulatory reports and responding to ad-hoc data analysis requests.
- Conducting code reviews for data engineers and collaborating with team product owners to define, scope and plan projects.