Senior Data Engineer
Skills
Software DevelopmentAmazon KinesisAmazon S3Amazon Web ServicesAnalytical SkillsApache KafkaData Engineering
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- Bachelor's degree or equivalent relevant experience
- 6-8 years of industry experience in data engineering or software engineering with a focus on data systems
- Strong proficiency in Python and SQL for data pipeline development and analysis
- Experience with AWS data platforms Glue S3 Athena Lambda Step Functions
- Familiarity with Snowflake or similar cloud data platforms
- Familiarity with streaming technologies such as Kafka or Kinesis
- Experience with workflow orchestration tools Airflow Step Functions or similar
- Proficiency in infrastructure-as-code tools like Terraform
- Understanding of data modeling principles
- Excellent written and verbal communication with a strong collaborative focus
- Experience with CI/CD tailored for machine learning systems
Nice to have
- Insurance domain experience
- Regulated environment experience
- ML infrastructure familiarity
- Feature store experience
- Data governance experience
- Cataloging tools exposure
Day to day
- Design and build robust data pipelines that integrate data from diverse sources to support risk evaluation and business intelligence.
- Build streaming data pipelines using Kafka and AWS services to enable real-time data processing and decision-making.
- Create and operate data services that make curated datasets accessible to internal teams and external partners through APIs and scheduled deliveries.
- Support data science workflows by building infrastructure for model training feature engineering and production inference.
- Implement data quality frameworks including validation monitoring and alerting to maintain trust in data assets.
- Contribute to CI/CD pipelines and infrastructure-as-code practices to ensure reliable repeatable deployments.
- Deliver highly reliable services by following engineering best practices and participating in an on-call rotation to support production systems.
- Contribute to the data warehouse by creating well-modeled datasets in Snowflake that empower analysts and data scientists.
- Collaborate with data scientists analysts and business stakeholders to understand requirements and deliver solutions that drive value.