Data Engineer
Skills
Computer ScienceApache SparkComplex SystemsDownstream Oil And GasDriftEnhanced Data Rates For GSM EvolutionHbase
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- 2+ years building and operating production software systems
- Fluency in at least one programming language
- Experience debugging moderately complex systems and improving reliability and performance
- Strong fundamentals in data structures testing version control and Linux basics
- Spark or PySpark experience
- Hadoop ecosystem exposure including HDFS or HBase
- Workflow orchestration experience with Airflow Dagster or NiFi
- Search and indexing experience with OpenSearch or MongoDB
- Kubernetes and infrastructure as code experience
- Degree in Computer Science or a numerical degree
Nice to have
- Pragmatic
- Collaborative
- Detail-oriented
- Reliability-focused
- Analytical
Day to day
- Engineer distributed ingestion services that reliably pull data from diverse sources and deliver clean structured outputs to downstream products
- Build high throughput batch and near real time processing components with a focus on performance scalability and predictable cost
- Design and evolve data contracts with schemas validation rules versioning and backward compatibility while adding observability and hardening reliability