Data Engineer

Skills

Azure Data LakeData EngineeringApache NifiAzure DatabricksData LoadingData ManipulationData Pipelines

Role

What the job involves

The main requirements, responsibilities and hiring steps.

Proficiency in NiFi with experience in creating and optimizing processors and process groups
Strong knowledge of Azure Datalake with experience in handling Delta Lake file formats (Parquet)
Expertise in Python and PySpark for data manipulation and analysis
Advanced SQL skills with the ability to work across multiple databases including Oracle, SQL Server, and proprietary systems
Experience with Databricks for data engineering tasks, including building and managing data pipelines
Experience in building pipeline strategies for data loading and ETL processes, including setting up and managing workflows.

Develop and enhance data processing systems to ensure seamless data integration across multiple databases including Oracle, SQL Server, and proprietary databases
Implement and optimize NiFi processors and process groups to streamline data workflows and enhance performance
Design, develop, and maintain ETL pipelines using Azure Datalake with Delta Lake file format (Parquet) to support data analytics and reporting
Perform SQL queries to extract, transform, and load data from various sources into target systems ensuring data integrity and consistency
Collaborate with cross-functional teams to identify and resolve system issues, ensuring data quality and reliability.