RRemoteville
Novi Labs logoNovi Labs

Senior Data Engineer

Skills
Data EngineeringETL ToolsAmazon Web ServicesApache AirflowApache SparkData WarehousingDatabases
Role

What the job involves

The main requirements, responsibilities and hiring steps.

Requirements

  • Small-company experience as FT employee only
  • Degree in Computer Science, Engineering, or related field preferred
  • Minimum of 7 years of experience in data engineering or related role
  • Proficiency in Apache Airflow for workflow orchestration
  • Expertise in Apache Spark for distributed data processing
  • Extensive experience with ETL tools and large-scale data processing
  • Strong programming skills in Python
  • Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud
  • Proficiency with SQL and experience with relational and NoSQL databases
  • Experience with containerization (e.g., Docker) and orchestration (e.g., Kubernetes)
  • Understanding of data warehousing and experience with modern data warehouses
  • Experience with infrastructure-as-code tools (e.g., Pulumi, Terraform)
  • Familiarity with CI/CD pipelines and DevOps practices in data engineering
  • Excellent problem-solving and analytical skills
  • Strong communication and collaboration skills

Nice to have

  • Knowledge of Oil & Gas data
  • Experience at a company delivering data product
  • Knowledge of data visualization tools
  • Experience with machine learning pipelines

Day to day

  • Design, develop, and optimize scalable data pipelines and ETL processes to handle large volumes of data from multiple sources
  • Collaborate closely with data scientists, analysts, and other engineers to ensure efficient data flow and integration across various platforms
  • Provide technical leadership and contribute to the strategic development of our data infrastructure