Senior/Lead Data Engineer

Job Title – Senior/Lead Data Engineer Location: Hyderabad/Bangalore(Hybrid) Job Type : Fulltime • * Immediate joiners or within 15 days** Job Description: We are seeking an experienced Data Engineer Senior/Lead to design, develop, and maintain scalable data solutions on Azure and Databricks as part of our enterprise data modernization initiatives. The ideal candidate will have a strong background in data pipeline development, data integration frameworks, and cloud-based data engineering, with deep expertise in tools such as Databricks, Azure Data Factory, Alteryx, Ab Initio, Talend, and Informatica. This role will lead the design and delivery of high-performance, governed data architectures for large-scale enterprise clients, driving data reliability, compliance, and analytics readiness. Key Responsibilities • Design and implement scalable and reusable data pipelines using Databricks (PySpark, Delta Lake) and Azure Data Factory, ensuring high throughput and low latency. • Develop and maintain Common Data Platforms (CDP) and Medallion Architecture-based data lakes to consolidate legacy systems and improve data availability. • Create and manage backend data services using Python, Spark, or C#, ensuring performance, reliability, and scalability. • Develop data ingestion frameworks using metadata-driven and schema-drift–handling designs to support multi-source data integration. • Lead data integration and migration projects using Talend, Ab Initio, Alteryx, and Informatica, modernizing traditional ETL workflows into cloud-native pipelines. • Implement data quality, lineage, and governance frameworks (Unity Catalog, Data Stewardship, Data Catalog, MDM). • Design and maintain GDPR/CCPA-compliant data processing and ensure adherence to global data privacy standards. • Build and optimize Feature Store Data Marts to support AI/ML analytics teams and downstream BI consumers. • Collaborate with cross-functional teams (Data Scientists, BI Developers, Product Owners) to define and deliver data-driven business outcomes. • Mentor junior engineers in best practices of data engineering, CI/CD, and infrastructure-as-code deployments. • Utilize Redis and caching mechanisms to optimize performance and reduce load latency in analytic workloads. • Take end-to-end ownership of assigned modules ensuring timely delivery, documentation, and performance optimization. Technical Environment • Cloud Platforms: Microsoft Azure (Data Factory, Synapse, Data Lake, Databricks), AWS (S3, EC2, Glue, Redshift) • Data Engineering Tools: Databricks (DLT, Auto Loader, Unity Catalog), Talend, Ab Initio, Informatica, Alteryx • Programming Languages: Python, PySpark, SQL, Unix Shell Scripting • Data Modeling: Dimensional, Relational, and Medallion architecture • DevOps / CI-CD: GitHub Actions, Azure DevOps, Databricks Asset Bundles (DABs) • Data Governance & Security: RBAC/ABAC, Data Stewardship, MDM, GDPR/CCPA compliance • Monitoring & Scheduling: Airflow, Logic Apps, Azure Monitor Required Qualifications • 10+ years of experience in Data Engineering, Data Architecture, or ETL Development. • 5 plus years’ Experience or familiarity using dbt. • Proven experience leading Azure + Databricks–based modernization or migration programs. • Deep understanding of data integration tools (Talend, Informatica, Ab Initio, Alteryx) and metadata-driven pipeline design. • Hands-on experience with PySpark, SQL, and Azure Data Services. • Strong understanding of data modeling, data quality, and governance frameworks. • Experience with real-time and batch data ingestion, including Kafka, Kinesis, or Event Hub. • Excellent problem-solving, communication, and stakeholder management skills. Preferred Qualifications • Bachelor’s degree in computer science, or a related field • Strong analytical and problem-solving skills • Azure Data Engineer Certification or equivalent • Familiarity with AI/ML data preparation and Feature Store design • Experience implementing cost optimization and cluster governance on Databricks Why Join Us • Be part of a data modernization journey with a global enterprise brand. • Work with cutting-edge cloud and data engineering tools. • Collaborate with a highly skilled, cross-functional team of architects, analysts, and data scientists. • Opportunity to lead and innovate within a large-scale, multi-domain data transformation initiative.