Data Engineer
Skills
Azure Data FactoryComputer ScienceData AnalyticsData ServicesDatabasesDriftGlue
What the job involves
The main requirements, responsibilities and hiring steps.
Requirements
- Master’s degree in Computer Science Data Engineering or related field preferred
- Six years of hands-on data engineering experience building production-grade pipelines for Data Science and AI in multi-cloud environments or equivalent combination of education and experience
- Expert-level proficiency in Snowflake architecture including data sharing performance tuning and integration with external cloud AI services
- Advanced hands-on knowledge of AWS data services such as S3 Glue and Lambda and Azure data services such as Data Factory and Synapse
- Mastery of Python SQL and PySpark
- Deep experience with data orchestration and containerization including Docker
- Ability to interface with on-premise SQL Mainframe extracts and flat files and transform them for modern cloud consumption
- Strong understanding of Machine Learning feature engineering and Generative AI vectorization and embedding pipelines
- Ability to work in a team environment
- Ability to meet or exceed Performance Competencies
Nice to have
- get-it-done attitude
- problem solving
- judgment
- troubleshooting
- analysis
- discretion
Day to day
- Designs and maintains resilient ETL and ELT pipelines across on-premise systems and multi-cloud platforms to centralize data for analytics and AI use cases.
- Builds feature stores and optimized datasets that are clean versioned reproducible and statistically valid for machine learning teams.
- Develops data workflows for generative AI including extraction transformation chunking and loading into vector databases while improving performance governance and reliability.
- Partners with IT security DBA and infrastructure teams to solve connectivity access and approval challenges and to deliver production-ready integrations.