Senior Data Engineer

Key Responsibilities :

• Design, build, and maintain robust and scalable data pipeline architectures.

•  Assemble large, complex datasets that meet both functional and non-functional business requirements.

•  Identify, design, and implement internal process improvements — including automation of manual workflows, optimization of data delivery, and re-architecting infrastructure for greater scalability and reliability.

•  Design, build, and optimize ETL infrastructure to enable scalable, high-quality data workflows across diverse sources, leveraging SQL and modern data processing frameworks.

•  Build analytics tools that utilize the data pipeline to deliver actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.

•  Collaborate with stakeholders across Executive, Product, Data, and Design teams to resolve data-related technical issues and ensure their data infrastructure needs are met.

•  Ensure data integrity, separation, and security across multiple data centers and AWS regions.

•  Create data tools and frameworks to empower analytics and data science teams in building and optimizing products that drive innovation and establish market leadership.

•  Lead and mentor a small team of data engineers, fostering a culture of technical excellence, collaboration, and continuous improvement.

•  Provide technical guidance, set coding standards, conduct code reviews, and support career development for team members.

•  Work closely with data and analytics experts to continually enhance the functionality, reliability, and scalability of our data systems.

Data Engineering and Infrastructure:

•  6+ years of experience in a Data Engineering role, designing, building, and managing scalable and reliable data systems.

•   Proficient with big data and stream-processing technologies such as Spark and Kafka.

•   Hands-on experience with cloud platforms, particularly AWS services like EC2 and RDS.

•   Skilled in building and orchestrating data pipelines using tools like Airflow.

•   Experience with Databricks for scalable data processing and advanced analytics.

•   Strong knowledge of SQLMesh for modern data workflow management.

•   Extensive experience integrating and working with external data sources via REST APIs, GraphQL endpoints, and SFTP servers.

•  Strong communication skills and leadership capabilities are required.

Databases and Data Management: 

•   Expertise with relational and NoSQL databases, including Postgres and MongoDB.

•   Solid understanding of data modeling, data governance, and data security best practices.

Programming and Development:

•   Proficient in Python for data engineering, automation, and workflow scripting.

•   Familiarity with software engineering best practices, including version control, testing, and CI/CD pipelines for data workflows.

•   Experience with JavaScript and TypeScript is a plus. 

Analytics, Visualization, and BI: 

•  Skilled in implementing and supporting self-service BI tools to enable business teams with accessible, actionable insights.

•  Experience with Streamlit for building interactive data visualizations is a plus. 

Blockchain and Financial Data Expertise: 

•  Knowledge of blockchain technology and the cryptocurrency ecosystem is a nice-to- have, with a strong interest in staying up to date with emerging trends.

•  Experience working with financial datasets and financial engineering concepts is considered a strong advantage.

Our Stack:

We work with a modern and evolving technology stack, including but not limited to: 

• Cloud Infrastructure: AWS for cloud services and infrastructure management

•  Databases: PostgreSQL for relational data, MongoDB for non-relational (NoSQL) data, and Redis for caching and real-time data management

•  Backend: NestJS (Node.js, TypeScript) and Python for building scalable backend services

•  Frontend: React for web applications, Streamlit for building interactive data visualizations

•  Data Engineering: Airflow and SQLMesh for data pipeline orchestration and modern workflow management

• Big Data & Processing: Databricks and Kafka for scalable data processing, analytics, and streaming

• Integrations & APIs: Extensive use of REST APIs, GraphQL, SFTP, and Slack integrations to enable seamless data exchange and operational workflows

Messaging&EventStreaming: Kafkaforreal-timedatapipelinesandevent-driven architectures