Databrick Engineer

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.

• Build and optimize ETL/ELT processes for ingesting data from multiple source systems.

• Develop data transformation logic and implement business rules using Spark frameworks.

• Work closely with business stakeholders, data analysts, and architects to understand data requirements.

• Implement and manage Delta Lake architecture for reliable and high-performance data processing.

• Optimize Databricks jobs, notebooks, clusters, and Spark code for performance and cost efficiency.

• Develop and support data ingestion frameworks using Azure Data Factory (ADF) and other integration tools.

• Monitor production workloads, troubleshoot failures, and ensure data availability.

• Implement data quality checks, validations, and reconciliation processes.

• Ensure adherence to data governance, security, and compliance standards.

• Participate in code reviews, technical design discussions, and best practice implementation.

• Support production deployments and provide L2/L3 support for critical data pipelines.