Senior Data Engineer - Senior Manager

Role: Senior Developer – Modern Data Platform Architect

Role Summary

We are seeking a highly skilled Senior Developer / Data Platform Engineer to design and build scalable data pipelines and modern Lakehouse solutions on Databricks-based platforms. The candidate should have strong expertise in PySpark, ETL/ELT frameworks, Data Quality, Apache Iceberg, and job orchestration, with exposure to asset management domain considered a strong advantage.

Key Responsibilities

1. Data Platform Development & Engineering

• Design and develop scalable data ingestion and transformation pipelines using PySpark and SQL.
• Build and manage batch and streaming ETL/ELT pipelines on Databricks.
• Implement data processing frameworks supporting high-volume enterprise datasets.

2. Lakehouse Architecture & Iceberg Implementation

• Design and maintain modern data Lakehouse architecture (Bronze/Silver/Gold layers).
• Work on Apache Iceberg / open table formats including schema evolution, partitioning, and compaction strategies.
• Ensure interoperability across engines and optimize data storage formats for performance.

3. Data Ingestion & Pipeline Management

• Design and monitor end-to-end ingestion pipelines (batch + near real-time).
• Build reusable ingestion frameworks for structured and semi-structured data.
• Manage dependencies, data lineage, and pipeline orchestration workflows.

4. Data Transformation & dbt (Nice to Have)

• Implement transformation logic using dbt or similar ELT frameworks.
• Standardize transformation pipelines for maintainability and versioning.

5. Data Quality & Observability

• Implement robust data quality checks, validation rules, and reconciliation processes.
• Develop monitoring frameworks for pipeline failures, SLA tracking, and alerting.
• Ensure data governance and audit compliance across datasets.

6. Job Orchestration & Workflow Management

• Implement and manage orchestration tools (Airflow / Databricks Jobs / other schedulers).
• Handle pipeline scheduling, retries, dependency chaining, and failure recovery.

7. CI/CD & DevOps Enablement

• Build and maintain CI/CD pipelines for data workflows and infrastructure.
• Automate deployment of Databricks jobs, notebooks, and configurations.
• Manage code versioning, release management, and environment promotion.

8. Performance Optimization

• Optimize Spark jobs for cost, performance, and scalability.
• Tune queries, partitioning strategies, and resource utilization.

9. Collaboration & Stakeholder Engagement

• Work with business teams, data architects, and upstream/downstream systems.
• Translate business requirements (especially in asset management domain) into technical solutions.
• Support data modeling (Data Vault / Star Schema exposure is a plus).

10. Platform Governance & Best Practices

• Follow enterprise standards for data security, access control, and governance.
• Implement Unity Catalog / RBAC / metadata management best practices.
• Ensure compliance with platform architecture and operational standards.

Required Skills

Must Have

• Strong experience in Databricks and PySpark
• ETL/ELT pipeline design and development
• Data ingestion and pipeline orchestration
• SQL and distributed data processing
• Data quality and validation frameworks
• Experience with Apache Iceberg / open table formats
• Cloud experience (AWS preferred)

Good to Have

• Exposure to Asset Management / Financial Data domain
• dbt (Data Build Tool)
• Data modeling (Data Vault / Star Schema)
• Knowledge of Snowflake / Unity Catalog
• Experience with CI/CD (Terraform, Git-based deployments)

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at StateStreet.com/careers

Read our CEO Statement