דרושים » דאטה » Data Engineer

Required Data Engineering Manager
Description
We are one of the most popular and downloaded apps in the world. Working with us provides a unique opportunity to influence hundreds of millions of our users and to be part of the journey that makes us a super-app. Our mission is to make peoples lives easier by enabling meaningful connections, from precious moments with family and friends, through managing business relationships to pursuing their passions.
As an Engineering Manager in the data department, youll build and scale our data platform and data apps that powers our business insights. Youll design and implement robust pipelines to process billions of daily records, leveraging cutting-edge cloud technologies to transform data into actionable intelligence.
If youre passionate about data engineering and driving business growth through insights, wed love to hear from you!
Responsibilities
Lead and grow Data Engineering and Machine Learning teams in a high-scale environment (tens of billions of events per day).
Own the design and evolution of a self-service data platform enabling internal teams to easily build, ship, and consume data products.
Architect and scale batch and streaming pipelines powering core business and ML use cases.
Drive production ML systems end-to-end (recommendation, ranking, prediction) with direct business KPI impact.
Ensure reliability, scalability, and observability of large-scale data and ML systems in production.

We're looking for a Data Platform Engineer to own and scale the Kubernetes infrastructure powering our large-scale data processing platform.

This is a hands-on role at the intersection of infrastructure and data engineering. You'll operate Kubernetes clusters running thousands of nodes, supporting workloads like Spark, Airflow, and remote shuffle services. Your focus: making distributed data workloads reliable, cost-efficient, and performant at scale.

This is not a traditional DevOps or SRE role. You won't be building CI/CD pipelines or managing web services. Instead, you'll be deep in Spark executor scaling, shuffle optimization, batch scheduler tuning, and capacity planning for clusters that process massive datasets daily.

If you've tuned Spark on Kubernetes at scale, wrestled with shuffle storage bottlenecks, or optimized batch scheduling across thousands of concurrent pods - this role is for you.

WHAT YOU'LL DO:
Operate and scale Kubernetes clusters with thousands of nodes supporting large-scale Spark and data processing workloads.
Manage and optimize Apache Spark on Kubernetes - executor autoscaling, driver scheduling, resource tuning, spot instance strategies.
Deploy and tune remote shuffle services (e.g., Apache Celeborn) to handle shuffle data at scale across multiple availability zones.
Operate and improve self-hosted Apache Airflow infrastructure on Kubernetes
Configure and optimize batch schedulers (e.g., YuniKorn, Volcano) for gang scheduling, fair-share queuing, and resource prioritization.
Drive cost optimization across large compute fleets - spot vs. on-demand strategies, node right-sizing, autoscaling policies, local SSD utilization.
Support and collaborate with Data Engineering teams on workload. performance, resource allocation, and infrastructure requirements.
Manage infrastructure-as-code (Terraform) and GitOps deployments (ArgoCD, Helm) for data platform services.
Integrate with managed data platforms (e.g., Databricks) and cloud storage for hybrid processing architectures.

The AI Engineering group builds modern infrastructure and solutions that improve how algorithms are developed at our company.
We are a small, independent team of experienced engineers with a mix of skills in algorithms, software, and infrastructure. We work in a DevOps style and build cross-team solutions that support research and development of advanced perception algorithms.
Our flagship project is a unified AV dataset used to train and evaluate next-generation models. We take large volumes of multi-camera video, object labels, HD maps, and sensor data from across the organization, and turn it into a curated, high-quality training set - at scale.
We are looking for someone who brings ML and computer-vision depth to the team - someone who can help shape the intelligence layer that decides what data is worth training on.
What will your job look like:
Work collaboratively with shared ownership. Your focus area will be the curation and ML side of our data pipeline, but you will contribute across the full stack alongside the rest of the team.
Build and improve the curation pipeline - from vision-model embeddings and scene detection, through VLM-based scene analysis, to scoring, deduplication, and sampling that produces a balanced and diverse dataset.
Run and optimize GPU inference at scale (embedding extraction, VLM inference) across thousands of driving sessions using workflow orchestration.
Develop scoring and sampling strategies that ensure rare but important scenarios (night driving, adverse weather, hazardous situations) are well-represented in the final dataset.
Work with algorithm teams to understand what data gaps hurt model performance and translate those into curation criteria.
Build validation and diagnostics that measure dataset quality - not just pipeline health, but whether the data is actually good for training.
Contribute to the core dataset SDK, converter, and 3D-geometry tooling (camera projection, calibration, coordinate transforms).