Data Engineer – SAS & PySpark

Job Description – Data Engineer – SAS & PySpark

Project Overview

We are seeking a Data Engineer to support a Data Catalogue Upgrade project, focusing on modernizing metadata, data lineage, and data governance capabilities. The role is hands-on and delivery-focused, with PySpark as the primary technology and SAS as a source system for data migration and integration.

Key Responsibilities

• Design, develop, and optimize PySpark-based data processing pipelines for large-scale datasets
• Migrate and transform data and metadata from SAS environments into the upgraded Data Catalogue platform
• Develop and maintain SQL queries for data validation, reconciliation, and quality checks
• Support data lineage, metadata enrichment, and data classification as part of the catalogue upgrade
• Collaborate with data architects, governance teams, and business stakeholders to ensure accurate metadata representation
• Troubleshoot and resolve data processing, performance, and integration issues
• Contribute to documentation, best practices, and knowledge transfer activities

Required Skills & Experience

• 5+ years of strong hands-on experience with PySpark (core skill)
• Solid experience working with SAS (e.g., SAS datasets, procedures, or migrations)
• Strong SQL skills for querying, validation, and transformation
• Experience in data engineering, ETL/ELT pipelines, or data platform modernization projects
• Understanding of data catalogues, metadata management, or data governance concepts

Nice to Have

• Experience with Data Catalogue or Metadata Management tools
• Exposure to cloud-based data platforms (AWS, Azure, or GCP)
• Knowledge of data lineage, data quality frameworks, or regulatory reporting
• Prior experience in large-scale data migration or modernization initiatives

Job Type: Full-time

Pay: Up to ₹80,000.00 per month

Benefits:

• Health insurance
• Paid sick time
• Paid time off
• Provident Fund
• Work from home

Work Location: Remote