RRemoteville
Metapack logoMetapack

Senior Data Engineer

Skills
AirflowAthenaBig DataData ArchitectureData ModelingDatabasesDatasets
Role

What the job involves

The main requirements, responsibilities and hiring steps.

Requirements

  • A Software Engineering background
  • Experience developing and supporting robust, automated and reliable data pipelines in Python and SQL
  • Experience with data processing frameworks like Pandas, Polars, dbt or Spark
  • Experience with streaming data processing
  • AWS, Azure or Google Cloud experience
  • Continuous integration/delivery environment experience with a passion for automation
  • Knowledge of a data orchestration solution like Airflow, Oozie, Luigi or Prefect
  • Knowledge of both relational and non-relational database design and internals
  • Knowledge of how to design distributed systems and the trade-offs involved
  • Experience working with software engineering best practices for development, including source control systems, automated deployment pipelines like Jenkins and devops tools like Terraform

Nice to have

  • MLOps know-how
  • Fast-learner
  • Interest in evaluating new technologies
  • Willing to attend conferences, webinars and meet-ups

Day to day

  • Contribute to the design, build and operational management of our data lake and analytics solution on top of proven AWS data technologies like S3, Athena, Lambda, Kinesis and ECS
  • Use state of the art technologies like Airflow, dbt and Spark to process data and get our dataset just right
  • Develop frameworks and solutions that enable us to acquire, process, monitor and extract value from our massive dataset
  • Support the Data Analysts and Data Scientists with automation, tooling, data pipelines and data engineering expertise, enabling them to deliver complex analytics products and machine learning solutions
  • Deliver highly reliable software and data pipelines using Software Engineering best practices like automation, version control, continuous integration/continuous delivery, testing, security, etc
  • Define, implement and enforce automated data security and data governance best practices within the solutions designed
  • Mentor more junior colleagues and receive mentorship by more senior colleagues

Hiring process

  • Salary discussion in first interview