RRemoteville
Hack The Box logoHack The Box

Senior Data Engineer

Skills
Software As A ServiceApache BeamApache SparkData ServicesDriftModelingMusic Composition
Role

What the job involves

The main requirements, responsibilities and hiring steps.

Requirements

  • Strong data modelling and warehouse architecture skills
  • Hands-on experience with GCP data services especially BigQuery
  • Production experience with streaming pipelines on Dataflow Beam Flink or Spark Structured Streaming
  • Solid SQL and strong Python for production-quality code
  • Experience with ClickHouse or another columnar OLAP engine
  • Workflow orchestration experience with Airflow or similar
  • Comfortable with dbt or equivalent transformation frameworks
  • Experience migrating off legacy warehouses onto cloud-native stacks is a plus
  • Working knowledge of ML in production including feature engineering feature stores model deployment drift monitoring and retraining
  • Docker and Kubernetes experience
  • CI/CD mindset infrastructure-as-code sensibility and a bias for simple observable systems
  • Bonus: CDC tooling such as Datastream or Debezium and Vertex AI or Feature Store

Nice to have

  • Opinionated about clean architecture
  • End-to-end ownership
  • Pragmatic
  • Low-overengineering bias
  • Collaborative

Day to day

  • Own and evolve data pipelines on GCP across streaming and batch workloads
  • Design and build ELT and ETL workflows that clean model and serve trustworthy data for analytics ML training and online inference
  • Partner with ML engineers to build feature pipelines monitor drift and support retraining while integrating APIs and third party data sources

Hiring process

  • Submit application
  • Meet Talent Acquisition team
  • Meet hiring team
  • Complete 2 assignments and debrief
  • Final conversation with senior leadership
  • Receive and accept offer