METONEUM

Data Engineer

Skills

APIAnalytical SkillsApache KafkaApache SparkBusiness ProcessData FeedsData Storage

Role

What the job involves

The main requirements, responsibilities and hiring steps.

Combine data from diverse sources including databases APIs and external data feeds into a unified system
Cleanse and preprocess data for accuracy and consistency before analysis or reporting
Develop and manage database schemas including relational and non-relational databases
Monitor and optimize database performance for efficient data retrieval and storage
Create and maintain data warehouses for large-scale analysis and reporting
Create and implement data models representing business processes and data relationships
Work with Hadoop Spark and Kafka for large-scale data processing and real-time streaming
Implement security measures to protect sensitive data and comply with regulations
Control access to data systems so only authorized users can access or manipulate data
Collaborate with data scientists to support analytical requirements
Continuously monitor data systems for performance issues errors or failures
Maintain comprehensive documentation for data pipelines schemas and data models
Create reports on data processing activities system performance and data quality

Design and develop robust data pipelines to extract transform and load data from diverse sources into reliable storage solutions.
Combine data from databases APIs and external feeds into a unified system that supports analysis and reporting.
Cleanses preprocesses and monitors data systems while optimizing performance security and documentation for stakeholders.