Data Engineer
Skills
Business IntelligenceData WarehousingIdentity And Access ManagementInformaticaManagement Information SystemsModelingPyspark
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- Active SC Security Clearance
- Proven experience delivering data engineering solutions within UK Central Government
- Strong commercial experience with Informatica
- Hands-on expertise with AWS services including Amazon EMR S3 and Lambda
- Advanced PySpark Python and SQL skills for large-scale data processing
- Experience building and maintaining enterprise-scale batch data pipelines
- Strong understanding of cloud-native data engineering architectures
- Experience modernising legacy ETL platforms and migrating workloads to AWS
- Experience implementing CI/CD pipelines Infrastructure as Code Terraform and automated testing
- Strong knowledge of Git GitLab branching strategies and code review practices
- Experience designing and consuming APIs for secure data integration
- Strong understanding of data quality governance and engineering best practices
- Experience working in Agile delivery teams using Jira and Confluence
- Excellent stakeholder management skills with the ability to communicate effectively across both technical and non-technical audiences
- Experience producing clear technical documentation and contributing to architectural discussions
Nice to have
- Collaborative
- Inclusive
- Supportive
- Mentoring mindset
- Agile
- Quality-focused
Day to day
- Design, build, and maintain production-grade data pipelines using AWS services including Amazon EMR, S3 and Lambda
- Develop scalable data processing solutions using PySpark, Python and SQL with a focus on performance resilience testing and observability
- Lead the migration and modernisation of legacy ETL and reporting workloads into cloud-native architectures
- Develop reusable data engineering components and shared frameworks
- Build and optimise data transformations and models supporting MI reporting and analytics including downstream Power BI solutions
- Work with Informatica as part of enterprise data integration and migration activities
- Own engineering best practices including CI/CD automated testing version control and release governance
- Conduct code reviews and mentor engineers promoting high-quality engineering standards across delivery teams
- Collaborate with architects product owners and platform teams to translate business requirements into robust technical solutions
- Participate fully in Agile ceremonies using Jira and Confluence
- Embed security governance and compliance throughout the engineering lifecycle ensuring solutions meet UK Government security and data handling requirements