דרושים » דאטה » לארגון פיננסי דרוש /ה data ENGINEER

We're looking for a Data Platform Engineer to own and scale the Kubernetes infrastructure powering our large-scale data processing platform.

This is a hands-on role at the intersection of infrastructure and data engineering. You'll operate Kubernetes clusters running thousands of nodes, supporting workloads like Spark, Airflow, and remote shuffle services. Your focus: making distributed data workloads reliable, cost-efficient, and performant at scale.

This is not a traditional DevOps or SRE role. You won't be building CI/CD pipelines or managing web services. Instead, you'll be deep in Spark executor scaling, shuffle optimization, batch scheduler tuning, and capacity planning for clusters that process massive datasets daily.

If you've tuned Spark on Kubernetes at scale, wrestled with shuffle storage bottlenecks, or optimized batch scheduling across thousands of concurrent pods - this role is for you.

WHAT YOU'LL DO:
Operate and scale Kubernetes clusters with thousands of nodes supporting large-scale Spark and data processing workloads.
Manage and optimize Apache Spark on Kubernetes - executor autoscaling, driver scheduling, resource tuning, spot instance strategies.
Deploy and tune remote shuffle services (e.g., Apache Celeborn) to handle shuffle data at scale across multiple availability zones.
Operate and improve self-hosted Apache Airflow infrastructure on Kubernetes
Configure and optimize batch schedulers (e.g., YuniKorn, Volcano) for gang scheduling, fair-share queuing, and resource prioritization.
Drive cost optimization across large compute fleets - spot vs. on-demand strategies, node right-sizing, autoscaling policies, local SSD utilization.
Support and collaborate with Data Engineering teams on workload. performance, resource allocation, and infrastructure requirements.
Manage infrastructure-as-code (Terraform) and GitOps deployments (ArgoCD, Helm) for data platform services.
Integrate with managed data platforms (e.g., Databricks) and cloud storage for hybrid processing architectures.

We are building a centralized intelligence system for the entire organization - a system that collects, consolidates, and analyzes all organizational data, and actively drives business workflows and decision-making.
This role exists to take that system to the next level.
As the Lead Organizational Intelligence Engineer, you will own the evolution of an internal platform that already influences how decisions are made across the company - and turn it into a foundational capability that fundamentally reshapes how the organization operates. Youll work end-to-end: from data ingestion and modeling, through analytics and AI-driven insights, all the way to decision workflows used by executives, R&D, product, and go-to-market teams.
You will operate as a senior individual contributor with broad organizational influence, while laying the technical and product foundations for a future team.
What Youll Do:
Own the architecture and implementation of a company-wide organizational intelligence system, end-to-end
Design and build data pipelines that ingest and unify data from across the organization and into our own platform
Define and evolve data models and semantic layers that create a shared, consistent language across teams and domains
Build advanced analytics and insight mechanisms that move beyond dashboards into decision-enabling workflows
Work directly with company executives to co-define requirements, identify blind spots, and translate strategic questions into systems and capabilities
Apply a product-driven mindset: deciding not just how to build, but increasingly what should be built and why
Leverage frontier AI tooling and agent-based workflows to accelerate development, improve system quality, and expand whats possible
Write and own production-grade code, setting a high bar for quality, reliability, and scalability
Influence technical and organizational workflows across the company to enable deeper intelligence and better decisions
Lay the groundwork for future externalization of these capabilities into customer-facing products