Data Engineer
Skills
Cloud ComputingApacheArtificial IntelligenceData PipelinesEngineeringHTMLMachine Learning
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- 3–5 years of experience designing and operating production data pipelines at scale
- Strong proficiency in Python for data engineering tasks
- Hands-on experience with Apache Airflow DAG-based pipelines
- Expertise with Docker Kubernetes and Helm
- Practical experience with Dask or Apache Spark
- Familiarity with cloud deployment and management
- Experience writing data services in Go or Rust
- Hands-on with Google Cloud services such as Pub/Sub BigQuery Cloud Storage and GKE
- Exposure to Ray or similar ML training frameworks
- Familiarity with Terraform
- Knowledge of data governance encryption and role-based access control
Nice to have
- Scalable mindset
- Cross-functional collaboration
- Problem-solving ability
- Attention to reliability
- Cloud-first thinking
Day to day
- Design and build production-grade batch and streaming data pipelines for cyber-security use cases.
- Containerize and orchestrate data-processing workloads using Docker Kubernetes and Helm.
- Optimize distributed transformations and training workflows for performance resiliency and cost efficiency.
- Implement observability and reliability controls to keep pipelines healthy and SLA-compliant.
- Collaborate with data scientists and software engineers on data contracts schemas quality checks and best practices.