Atlas Systems - Senior QA Engineer - Microsoft Fabric

Job Title : Senior QA Engineer Microsoft Fabric (Data Engineering & Notebook Validation) Role Summary We are seeking an experienced Senior QA Engineer (7+ years) with strong expertise in Microsoft Fabric, particularly within the Data Engineering and Data Analytics environment. The Ideal Candidate Will Have Hands-on Experience In • Developing and testing notebooks in Fabric (PySpark / Spark SQL) • Supporting and validating data migrations to Microsoft Fabric • Designing automated data validation frameworks • Ensuring data integrity across Lakehouse, pipelines, and Power BI environments This role requires a technically strong QA professional who understands both backend data validation and automation within modern data platforms. Key Responsibilities Microsoft Fabric & Data Engineering QA : • Perform end-to-end validation of data pipelines within Microsoft Fabric. • Validate data across OneLake, Lakehouse, and Fabric SQL endpoints. • Test Bronze, Silver, and Gold data layers for completeness, accuracy, and transformation integrity. • Collaborate with Data Engineers to validate transformation logic and incremental load processing. Notebook Development & Validation • Develop and execute validation logic within Fabric notebooks using PySpark and Spark SQL. • Perform schema validation, row count reconciliation, null checks, duplicate checks, and aggregation validation. • Build reusable notebook-based validation frameworks. • Log validation results into Delta tables for auditability and traceability. Data Migration To Microsoft Fabric • Validate data migrations from legacy platforms (AWS, Azure, On-prem, etc.) to Microsoft Fabric. • Perform before-and-after data comparison between legacy and Fabric environments. • Validate transformation logic, metadata migration, and incremental load accuracy. • Support report-level validation post-migration (Power BI comparisons). QA Automation In Data Platforms • Design and implement automated validation frameworks for data pipelines. • Integrate notebook-based validations into Fabric pipelines. • Support CI/CD integration for automated QA gating. • Develop automated checks for : • Data completeness • Schema consistency • Business rule validation • Incremental load verification Collaboration & Reporting • Work closely with Data Engineers, BI Developers, and Business Analysts. • Perform root cause analysis for data discrepancies. • Document test cases, validation logic, and QA metrics. • Provide QA sign-off for production releases. Required Qualifications • 7+ years of experience in Data QA, ETL Testing, or BI/Data Engineering validation. • Strong hands-on experience in Microsoft Fabric (Data Engineering environment). • Proven experience writing and executing Fabric notebooks (PySpark / Spark SQL). • Experience supporting data migration projects to Microsoft Fabric. • Strong SQL skills for backend data validation. • Experience validating Delta/Parquet data structures. • Understanding of data quality principles : • Accuracy • Completeness • Consistency • Timeliness • Experience designing or implementing QA automation in data platforms. • Exposure to CI/CD pipelines and DevOps integration for data testing. Preferred Qualifications • Experience validating Power BI reports post Fabric migration. • Knowledge of OneLake architecture and Delta Lake format. • Experience in Azure Data ecosystem. • Exposure to Python scripting for data validation. • Experience in regulated or enterprise data environments. Key Technical Skills • Microsoft Fabric (Lakehouse, OneLake, Fabric SQL) • PySpark • Spark SQL • Advanced SQL • Delta Lake • Data migration validation • QA automation frameworks • CI/CD integration • Power BI validation (preferred) What Success Looks Like • Successful validation of Fabric migrations with zero critical data defects. • Automated notebook-based data validation framework in place. • Reduced manual data testing effort through automation. • High confidence in data accuracy across reporting and analytics layers. (ref:hirist.tech)