How Remoteville checks and expires listings
Senior Data Engineer
Skills
Data EngineeringETL ToolsAmazon Web ServicesApache AirflowApache SparkData WarehousingDatabases
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- Small-company experience as FT employee only
- Degree in Computer Science, Engineering, or related field preferred
- Minimum of 7 years of experience in data engineering or related role
- Proficiency in Apache Airflow for workflow orchestration
- Expertise in Apache Spark for distributed data processing
- Extensive experience with ETL tools and large-scale data processing
- Strong programming skills in Python
- Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud
- Proficiency with SQL and experience with relational and NoSQL databases
- Experience with containerization (e.g., Docker) and orchestration (e.g., Kubernetes)
- Understanding of data warehousing and experience with modern data warehouses
- Experience with infrastructure-as-code tools (e.g., Pulumi, Terraform)
- Familiarity with CI/CD pipelines and DevOps practices in data engineering
- Excellent problem-solving and analytical skills
- Strong communication and collaboration skills
Nice to have
- Knowledge of Oil & Gas data
- Experience at a company delivering data product
- Knowledge of data visualization tools
- Experience with machine learning pipelines
Day to day
- Design, develop, and optimize scalable data pipelines and ETL processes to handle large volumes of data from multiple sources
- Collaborate closely with data scientists, analysts, and other engineers to ensure efficient data flow and integration across various platforms
- Provide technical leadership and contribute to the strategic development of our data infrastructure