Data Engineering Technical Lead

Skills
AirflowData Build ToolGoogle BigqueryOptimisingPyspark
Role

What the job involves

The main requirements, responsibilities and hiring steps.

Requirements

  • 3+ Years operational experience of Software and Data Architecture Patterns
  • 3+ Years operational experience of Python Data Science Development
  • 3+ Years operational experience of Data Warehousing, Data Modelling, Data Transformation and Data Quality
  • 3+ Years operational experience of NoSQL Development (BigQuery / MongoDB)
  • 3+ Years operational experience of Distributed Data Warehousing, Query Optimisation, Data Modelling, Role / User Management, BigQuery Monitoring & Alerting
  • Operational experience of Data Reporting Tools, Apache Superset, Tableau, BI
  • Operational Experience of Apache Airflow, DAG Management, Airflow Monitoring & Alerting
  • Familiarity with Cloud APIs and SDKs, Unit Testing Strategies
  • Strong understanding of Agile/Scrum methodologies
  • Strong diagnostic/troubleshooting skills for complex enterprise systems
  • Able to quickly learn and work with new languages and technologies
  • Ability to build strong, collaborative working relationships
  • Open attitude to sharing ideas, knowledge, and information
  • Can work in a challenging, fast-paced environment

Day to day

  • Leads, designs, develops, and provisions data warehousing, data orchestration, data transformation, and ETL solutions aligning with industry best practices and team standards.
  • Manages the availability and performance of the data platform through monitoring tools, ensuring quick interventions for operational issues.
  • Establishes standards for technical delivery, ensuring best practices in version control, testing, and deployment.
  • Ensures data security compliance across multiple states and regions.
  • Leverages multi-cloud services for optimal data management and processing.
  • Develops strategies for disaster recovery and business continuity.
  • Designs and implements business intelligence solutions in collaboration with stakeholders.