Senior Data Engineer - Hybrid

Azure Databricks Data Engineer Experience: 6+ Years Fulltime We are looking for a Data Engineer to design and build scalable, production-grade data pipelines on Azure using Databricks. The role involves working with high-volume, high-velocity enterprise data across domains such as telecom, retail, and regulatory systems (e.g., GST, eWay Bill). You will be responsible for building reliable batch and real-time pipelines, ensuring data quality, auditability, and performance at scale. Data Engineering & Pipeline Development • Design and implement end-to-end data pipelines using Azure Databricks and PySpark • o Azure Data Factory (ADF) o Kafka / Azure Event Hubs • Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers) • Develop reusable data models for analytics, reporting, and downstream consumption • Ensure data lineage, traceability, and auditability across layers Delta Lake & Data Management • o MERGE (upserts), SCD Type 1/2 implementations o Late-arriving data Performance Optimization & Scalability • Troubleshoot performance bottlenecks such as: o Data skew Design orchestration workflows using Azure Data Factory: Data Quality & Governance • Implement robust data quality checks, including: o Data reconciliation across sources • Handle schema drift and evolving data contracts • Ensure compliance with regulatory and audit requirements o Regulatory datasets (GST, eWay Bill, financial reporting) o Retail / telecom data platforms • Ensure data consistency, reconciliation, and reporting accuracy Strong SQL: o Transformations and actions o Performance tuning Azure Data Factory (ADF) • Azure Data Lake Storage Gen2 (ADLS) • Kafka or Azure Event Hubs • CDC pipeline implementation • Data quality frameworks (e.g., Strong debugging skills in distributed data systems (Spark) • Experience handling production incidents and RCA (Root Cause Analysis) • Flexibility to work in WFO / hybrid setup Reliable, scalable pipelines handling large-scale enterprise data • Reduced pipeline failures and improved data SLAs • High-quality, trusted datasets for business-critical reporting • Efficient Spark jobs with optimized cost and performance