RRemoteville
Polestar Analytics logoPolestar Analytics

Databricks Architect

Skills
Azure DatabricksApache SparkPythonSQLScalaAzure Data FactoryApache Kafka
Role

What the job involves

The main requirements, responsibilities and hiring steps.

Requirements

  • Strong experience with Databricks platform
  • Expertise in Apache Spark Spark SQL and distributed computing
  • Programming experience in Python Scala or SQL
  • Experience with data lakes and lakehouse architectures
  • Strong experience with cloud data platforms such as Microsoft Azure
  • Strong experience with cloud data platforms such as Google Cloud Platform

Nice to have

  • Architectural thinking
  • Platform optimization mindset
  • Analytics domain knowledge
  • Collaborative approach
  • Problem-solving ability

Day to day

  • Design and implement modern data platforms using Databricks Lakehouse architecture.
  • Define enterprise data architecture data models and governance frameworks for analytics platforms.
  • Architect scalable batch and real-time pipelines and optimize Spark workloads cluster configurations and query performance.
  • Develop ETL and ELT pipelines using Spark Python SQL and cloud ingestion tools while enabling ML pipelines and feature engineering.