GCP Data Engineer

Job Description Key Responsibilities : Data Pipeline Development: Build and optimize ETL/ELT workflows using Airflow, Prefect, or similar orchestration tools. Data Integration & Transformation: Ingest data from multiple sources (transactional, behavioural, streaming) and prepare feature-ready datasets. Application Integration: Connect pipelines to Salesforce, Gainsight, and other enterprise applications for real-time data sync. BI Semantic Modelling: Design and maintain semantic layers for BI tools (Power BI, Looker, Tableau) to enable self-service analytics. Version Control & Experiment Tracking: Implement DVC for dataset versioning and integrate MLflow for experiment reproducibility. Distributed Data Processing: Use Spark, Ray, or Databricks for large-scale data transformations and feature engineering. Cloud Infrastructure: Deploy and manage pipelines on Azure Data Factory, Azure Synapse, GCP Dataflow, and Big Query; leverage storage (ADLS, GCS) and compute clusters. MLOps Enablement: Collaborate on CI/CD workflows for ML models, feature store integration, and monitoring pipelines. Data Quality & Governance: Implement validation, lineage tracking, and compliance checks for secure and reliable data. Performance Optimization: Profile and tune pipelines for cost efficiency and low latency. Collaboration: Partner with Data Scientists to ensure timely delivery of clean, well-structured data for ML/DL models. Required Skills : Technical: Python (pandas, PySpark), SQL; familiarity with Scala is a plus. Orchestration: Airflow, Prefect, DBT. Distributed frameworks: Spark, Ray, Databricks. Cloud platforms: Azure (Data Factory, Synapse, ADLS) and GCP (Dataflow, BigQuery, GCS). BI Semantic Modelling: Power BI, Looker, Tableau. Application Integration: Salesforce, Gainsight, REST APIs. Containerization: Docker; basic Kubernetes. Versioning: Git, DVC; experiment tracking: MLflow. Streaming: Kafka or Pub/Sub for real-time ingestion.