Senior ETL & Data Streaming Engineer

As a Senior ETL & Data Streaming Engineer at DataFlow Group, you will have the opportunity to play a crucial role in designing, developing, and maintaining robust data pipelines. Your expertise in both batch ETL processes and real-time data streaming technologies, along with hands-on experience with AWS data services, will be instrumental in ensuring the efficiency and effectiveness of our data operations. • *Key Responsibilities:** - Design, develop, and implement highly scalable, fault-tolerant, and performant ETL processes using industry-leading ETL tools such as Talend, Informatica, Apache Kafka, or AWS Kinesis. - Architect and build batch and real-time data streaming solutions to support immediate data ingestion and processing requirements. - Collaborate with data architects, data scientists, and business stakeholders to understand data requirements and translate them into efficient data pipeline solutions. - Ensure data quality, integrity, and security across all data pipelines and storage solutions. - Monitor, troubleshoot, and optimize existing data pipelines for performance, cost-efficiency, and reliability. - Develop and maintain comprehensive documentation for all ETL and streaming processes, data flows, and architectural designs. - Implement data governance policies and best practices within the Data Lake and Data Warehouse environments. - Mentor junior engineers and contribute to fostering a culture of technical excellence and continuous improvement. - Stay abreast of emerging technologies and industry best practices in data engineering, ETL, and streaming. • *Qualifications:** - 10+ years of progressive experience in data engineering, focusing on ETL, ELT, and data pipeline development. - Extensive hands-on experience with commercial ETL tools, particularly Talend. - Proven experience with real-time data ingestion and processing using platforms such as AWS Glue, Apache Kafka, AWS Kinesis, or similar. - Proficiency with AWS S3 for data storage and management. - Hands-on experience with AWS Glue for ETL orchestration and data cataloging. - Familiarity with AWS Lake Formation for building secure data lakes. - Good to have experience with AWS EMR for big data processing. - Strong background in traditional data warehousing concepts, dimensional modeling, and DWH design principles. - Proficient in SQL and at least one scripting language (e.g., Python, Scala) for data manipulation and automation. - Strong understanding of relational databases and NoSQL databases. - Experience with version control systems (e.g., Git). - Excellent analytical and problem-solving skills with a keen eye for detail. - Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences. As a Senior ETL & Data Streaming Engineer at DataFlow Group, you will have the opportunity to play a crucial role in designing, developing, and maintaining robust data pipelines. Your expertise in both batch ETL processes and real-time data streaming technologies, along with hands-on experience with AWS data services, will be instrumental in ensuring the efficiency and effectiveness of our data operations. • *Key Responsibilities:** - Design, develop, and implement highly scalable, fault-tolerant, and performant ETL processes using industry-leading ETL tools such as Talend, Informatica, Apache Kafka, or AWS Kinesis. - Architect and build batch and real-time data streaming solutions to support immediate data ingestion and processing requirements. - Collaborate with data architects, data scientists, and business stakeholders to understand data requirements and translate them into efficient data pipeline solutions. - Ensure data quality, integrity, and security across all data pipelines and storage solutions. - Monitor, troubleshoot, and optimize existing data pipelines for performance, cost-efficiency, and reliability. - Develop and maintain comprehensive documentation for all ETL and streaming processes, data flows, and architectural designs. - Implement data governance policies and best practices within the Data Lake and Data Warehouse environments. - Mentor junior engineers and contribute to fostering a culture of technical excellence and continuous improvement. - Stay abreast of emerging technologies and industry best practices in data engineering, ETL, and streaming. • *Qualifications:** - 10+ years of progressive experience in data engineering, focusing on ETL, ELT, and data pipeline development. - Extensive hands-on experience with commercial ETL tools, particularly Talend. - Proven experience with real-time data ingestion and processing using platforms such as AWS Glue, Apache Kafka, AWS Kinesis, or similar. - Proficiency with AWS S3 for data storage and management. - Hands-on experience with AWS Glue for ETL orchestration and data cataloging. - Familiarity with AWS Lake Formation for building secure data lakes. - Good to have experience with AWS EMR for big data process