Senior Data Engineer - Hybrid
Celebal Technologies
Azure Databricks Data Engineer Experience: 6+ Years
Fulltime
We are looking for a Data Engineer to design and build scalable, production-grade data pipelines on Azure using Databricks. The role involves working with high-volume, high-velocity enterprise data across domains such as telecom, retail, and regulatory systems (e.g., GST, eWay Bill).
You will be responsible for building reliable batch and real-time pipelines, ensuring data quality, auditability, and performance at scale.
Data Engineering & Pipeline Development
- Design and implement end-to-end data pipelines using Azure Databricks and PySpark
- o Azure Data Factory (ADF)
o Kafka / Azure Event Hubs
- Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers)
- Develop reusable data models for analytics, reporting, and downstream consumption
- Ensure data lineage, traceability, and auditability across layers
Delta Lake & Data Management
- o MERGE (upserts), SCD Type 1/2 implementations
o Late-arriving data
Performance Optimization & Scalability
- Troubleshoot performance bottlenecks such as:
o Data skew
Design orchestration workflows using Azure Data Factory:
Data Quality & Governance
- Implement robust data quality checks, including:
o Data reconciliation across sources
- Handle schema drift and evolving data contracts
- Ensure compliance with regulatory and audit requirements
o Regulatory datasets (GST, eWay Bill, financial reporting)
o Retail / telecom data platforms
- Ensure data consistency, reconciliation, and reporting accuracy
Strong SQL:
o Transformations and actions
o Performance tuning
Azure Data Factory (ADF)
- Azure Data Lake Storage Gen2 (ADLS)
- Kafka or Azure Event Hubs
- CDC pipeline implementation
- Data quality frameworks (e.g., Strong debugging skills in distributed data systems (Spark)
- Experience handling production incidents and RCA (Root Cause Analysis)
- Flexibility to work in WFO / hybrid setup
Reliable, scalable pipelines handling large-scale enterprise data
- Reduced pipeline failures and improved data SLAs
- High-quality, trusted datasets for business-critical reporting
- Efficient Spark jobs with optimized cost and performance
Posted 5 Mar 2026 · Listing from OnJob.io. Create a free profile to apply and see your AI match score.
Related Data & AI jobs
Hand-picked roles that match this listing on skills, category and location — each scored to your profile inside OnJob.