We are looking for a Senior Big Data Engineer with proven experience as a Backend Engineer who will play a critical role at one of our Big Data teams. As a leading member, you will take part in building real-time detection & transformation engines and handling petabytes of data on top of our cloud infrastructure. As well as building Backend components in our platform.
RESPONSIBILITIES
Work as a member of a senior Agile Scrum team to design, develop, and maintain large-scale data stream processing pipelines and backend components on top of Flink, Spark, K8S, and AWS.
Design, develop, and implement solutions to handle scaling with good performance & cost that are easy to maintain.
Solve a wide variety of complex problems on a high scale.
Collaborate with engineers across our R&D group and Product Managers to improve our platform.
Develop and implement components features and components in our platform to give internal and external users the ability to build data transformations and detection pipelines at scale.
Develop and implement data quality checks to ensure accuracy and completeness of data and work on externalizing monitoring capabilities to provide full visibility into all stages of data ingestion.
Optimize and tune performance of Flink and WH jobs to handle large volumes of data.
Stay up-to-date with the latest technologies and trends in big data processing and distributed computing.
Participate in code reviews and ensure adherence to coding standards and best practices.
Requirements: 7+ years of experience as a Backend Engineer
5+ years of development experience in Scala with functional programming paradigm.
3+ years of hands-on experience in Big Data engineering with a focus on Scala/Java, Spark/Flink/Kafka, and cloud architecture (EMR/K8S).
Experience with modern Data lakes/warehouses such as Snowflake and Databricks.
Deep technical expertise in distributed systems, stream processing, and data modeling of large data sets.
Proven track record of delivering high-quality, scalable, and secure systems in a fast-paced working environment.
Experience with data governance practices, data security, and performance & cost optimization containers, working with AWS services such as S3, EKS, and more.
Strong problem-solving skills and ability to work independently.
A team player with excellent communication skills.
B.Sc. in computer science or an equivalent.
Advantages:
Experience with Hadoop MapReduce.
Advantage for experience in effect systems such as ZIO.
Production experience working with SaaS environments.
Experience in data modeling.
This position is open to all candidates.
