דרושים » תוכנה » Big Data Infrastructure Engineer-2572

Shape the Future of Data - Join our mission to build the foundational pipelines and tools that power measurement, insights, and decision-making across our product, analytics, and leadership teams.
Develop the Platform Infrastructure - Build the core infrastructure that powers our data ecosystem including the Kafka events-system, DDL management with Terraform, internal data APIs on top of Databricks, and custom admin tools (e.g. Django-based interfaces).
Build Real-time Analytical Applications - Develop internal web applications to provide real-time visibility into platform behavior, operational metrics, and business KPIs integrating data engineering with user-facing insights.
Solve Meaningful Problems with the Right Tools - Tackle complex data challenges using modern technologies such as Spark, Kafka, Databricks, AWS, Airflow, and Python. Think creatively to make the hard things simple.
Own It End-to-End - Design, build, and scale our high-quality data platform by developing reliable and efficient data pipelines. Take ownership from concept to production and long-term maintenance.
Collaborate Cross-Functionally - Partner closely with backend engineers, data analysts, and data scientists to drive initiatives from both a platform and business perspective. Help translate ideas into robust data solutions.
Optimize for Analytics and Action - Design and deliver datasets in the right shape, location, and format to maximize usability and impact - whether thats through lakehouse tables, real-time streams, or analytics-optimized storage.
You will report to the Data Engineering Team Lead and help shape a culture of technical excellence, ownership, and impact.

We are seeking an experienced Solutions Data Engineer who possess both technical depth and strong interpersonal skills to partner with internal and external teams to develop scalable, flexible, and cutting-edge solutions. Solutions Engineers collaborate with operations and business development to help craft solutions to meet customer business problems.
A Solutions Engineer works to balance various aspects of the project, from safety to design. Additionally, a Solutions Engineer researches advanced technology regarding best practices in the field and seek to find cost-effective solutions.
Job Description:
Were looking for a Solutions Engineer with deep experience in Big Data technologies, real-time data pipelines, and scalable infrastructuresomeone whos been delivering critical systems under pressure, and knows what it takes to bring complex data architectures to life. This isnt just about checking boxes on tech stacksits about solving real-world data problems, collaborating with smart people, and building robust, future-proof solutions.
In this role, youll partner closely with engineering, product, and customers to design and deliver high-impact systems that move, transform, and serve data at scale. Youll help customers architect pipelines that are not only performant and cost-efficient but also easy to operate and evolve.
We want someone whos comfortable switching hats between low-level debugging, high-level architecture, and communicating clearly with stakeholders of all technical levels.
Key Responsibilities:
Build distributed data pipelines using technologies like Kafka, Spark (batch & streaming), Python, Trino, Airflow, and S3-compatible data lakesdesigned for scale, modularity, and seamless integration across real-time and batch workloads.
Design, deploy, and troubleshoot hybrid cloud/on-prem environments using Terraform, Docker, Kubernetes, and CI/CD automation tools.
Implement event-driven and serverless workflows with precise control over latency, throughput, and fault tolerance trade-offs.
Create technical guides, architecture docs, and demo pipelines to support onboarding, evangelize best practices, and accelerate adoption across engineering, product, and customer-facing teams.
Integrate data validation, observability tools, and governance directly into the pipeline lifecycle.
Own end-to-end platform lifecycle: ingestion → transformation → storage (Parquet/ORC on S3) → compute layer (Trino/Spark).
Benchmark and tune storage backends (S3/NFS/SMB) and compute layers for throughput, latency, and scalability using production datasets.
Work cross-functionally with R&D to push performance limits across interactive, streaming, and ML-ready analytics workloads.
Operate and debug object storebacked data lake infrastructure, enabling schema-on-read access, high-throughput ingestion, advanced searching strategies, and performance tuning for large-scale workloads.

About the Role:We are seeking an experienced Senior Data Engineer to join our dynamic data team. The ideal candidate will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure, ensuring the availability, reliability, and quality of our data. This role requires strong technical expertise, problem-solving skills, and the ability to collaborate across teams to deliver data-driven solutions.Key Responsibilities:

Design, implement, and maintain robust, scalable, and high-performance data pipelines and ETL processes.
Develop and optimize data models, schemas, and storage solutions to support analytics and machine learning initiatives.
Collaborate with software engineers and product managers to understand data requirements and deliver high-quality solutions.
Ensure data quality, integrity, and governance across multiple sources and systems.
Monitor and troubleshoot data workflows, resolving performance and reliability issues.
Evaluate and implement new data technologies and frameworks to improve the data platform.
Document processes, best practices, and data architecture.
Mentor junior data engineers and contribute to team knowledge sharing.

We are seeking an experienced Data Platform Engineer to join our Storage Analytics team. You will design and build data solutions that provide critical insights into storage performance and usage across Apple's entire device ecosystem.
Description:
Working with large-scale telemetry data from millions of Apple devices worldwide, you'll support multiple CoreOS Storage teams, including Software Update, Backup/Restore/Migration, Storage Attribution, and other storage domains.
Responsibilities
Design, build, and maintain scalable data processing infrastructure to handle large-scale telemetry from Apple's global device fleet
Develop highly scalable data pipelines to ingest and process storage performance metrics, usage patterns, and system telemetry with actionable alerting and anomaly detection
Build and maintain robust data platforms and ETL frameworks that enable CoreOS Storage teams to access, process, and derive insights from storage telemetry data
Engineer automated data delivery systems and APIs that serve processed storage metrics to various engineering teams across different storage domains

Were looking for an experienced and passionate Staff Data Engineer to join our Data Platform group in TLV as a Tech Lead. As the Groups Tech Lead, youll shape and implement the technical vision and architecture while staying hands-on across three specialized teams: Data Engineering Infra, Machine Learning Platform, and Data Warehouse Engineering, forming the backbone of Lemonades data ecosystem.

The groups mission is to build a state-of-the-art Data Platform that drives Lemonade toward becoming the most precise and efficient insurance company on the planet. By embracing Data Mesh principles, we create tools that empower teams to own their data while leveraging a robust, self-serve data infrastructure. This approach enables Data Scientists, Analysts, Backend Engineers, and other stakeholders to seamlessly access, analyze, and innovate with reliable, well-modeled, and queryable data, at scale.

In this role youll :
Technically lead the group by shaping the architecture, guiding design decisions, and ensuring the technical excellence of the Data Platforms three teams

Design and implement data solutions that address both applicative needs and data analysis requirements, creating scalable and efficient access to actionable insights

Drive initiatives in Data Engineering Infra, including building robust ingestion layers, managing streaming ETLs, and guaranteeing data quality, compliance, and platform performance

Develop and maintain the Data Warehouse, integrating data from various sources for optimized querying, analysis, and persistence, supporting informed decision-makingLeverage data modeling and transformations to structure, cleanse, and integrate data, enabling efficient retrieval and strategic insights

Build and enhance the Machine Learning Platform, delivering infrastructure and tools that streamline the work of Data Scientists, enabling them to focus on developing models while benefiting from automation for production deployment, maintenance, and improvements. Support cutting-edge use cases like feature stores, real-time models, point-in-time (PIT) data retrieval, and telematics-based solutions

Collaborate closely with other Staff Engineers across Lemonade to align on cross-organizational initiatives and technical strategies

Work seamlessly with Data Engineers, Data Scientists, Analysts, Backend Engineers, and Product Managers to deliver impactful solutions

Share knowledge, mentor team members, and champion engineering standards and technical excellence across the organization

We are seeking a Senior Automation Engineer to join our Big Data team, where we build cutting-edge, AI-driven applications that empower the worlds largest financial institutions in the fight against financial crime.
Youll be part of a dynamic group responsible for the analytics and developer experience foundations, enabling software developers, data engineers, and data scientists to deliver scalable, high-quality analytics solutions quickly. In this role, youll ensure the quality, reliability, and performance of our intelligent cloud-based products through robust automated testing.
How will you make an impact?
Design, develop, and maintain automated tests for backend systems and data pipelines.
Implement and manage a scalable testing framework to support continuous integration and delivery.
Create test plans, and maintain detailed documentation of testing processes, results, and quality metrics.
Collaborate with developers and product managers to understand system architecture and business requirements.
Participate in requirements analysis, technical reviews, and test planning.
Identify, document, and track defects, drive resolution with development teams.
Monitor production environments, investigate anomalies, and push for timely fixes.
Ensure data health and correctness across systems.
Continuously improve QA processes, tools, and best practices.
Advocate for quality across the development lifecycle.

our company's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with information and one another. Our products need to handle information at massive scale, and extend well beyond web search. We're looking for engineers who bring fresh ideas from all areas, including information retrieval, distributed computing, large-scale system design, networking and data storage, security, artificial intelligence, natural language processing, UI design and mobile; the list goes on and is growing every day. As a software engineer, you will work on a specific project critical to our companys needs with opportunities to switch teams and projects as you and our fast-paced business grow and evolve. We need our engineers to be versatile, display leadership qualities and be enthusiastic to take on new problems across the full-stack as we continue to push technology forward.
The Waze Engineering Productivity (EngProd) team is the multiplier for all of Waze engineering.
In this role, you will design, build, and own the foundational systems that empower developers to ship with speed, quality, and confidence. You will be maintaining systems and leading the charge on Waze's technical initiatives.
Waze is where people and technology meet to solve transportation challenges. It's a platform that empowers users to contribute road data and edit Waze maps to improve the way we move about the world. As the social navigation pioneer, Waze leverages mobile technology and a passionate global community to redefine expectations of todays maps.
Responsibilities
Develop and maintain back-end services and libraries, in the Java ecosystem, that form the support of the development environment.
Take ownership of components within technical initiatives, such as the company-wide migration to our company3 or the rollout of new Artificial Intelligence (AI)-powered developer tools.
Deploy and manage mission-critical services on our company Cloud Platform (GCP) using technologies like Kubernetes and Docker, ensuring high availability and performance.
Collaborate with engineering teams across Waze to understand their issues, gather requirements, and deliver solutions that make their workflows efficient.
Work with Java, Python and our company's internal tooling to select the right technology for the job.

This role provides the opportunity to lead end-to-end design & implementation of advanced server-side features using a wide range of technologies: Apache Spark, Apache Airflow, Scala, k8s, Node.js (JS/Typescript) with MongoDB, Redis, Kafka, Dockers and Flink (Big Data Stream Processing)

If you are up for the challenge and you have the XM factor, come and join us!

We are looking for a talented and dedicated ML & Big Data Analyst who will be a part of the Core Data group.
As part of this role, you will help us process and streamline rich and complicated datasets. Will explore and validate new data sources. Be part of the data science and data engineering force creating new features for our solution.
What does the day-to-day of a ML & Big Data Analyst at our company look like:
Apply your expertise in quantitative analysis and data mining to turn data into insights.
Conduct research and develop tools that will ensure the quality of our companys data and algorithms.
Utilize your deep understanding of our companys data in order to evaluate and improve novel algorithms produced by the data scientists in your team.
Partner with Big Data Engineers to establish an infrastructure to scale solutions.

We are seeking a passionate and highly skilled Senior Golang Developer to join our dynamic Software Engineering team. The ideal candidate is an expert Go engineer with a strong background in building scalable, high-performance distributed backend systems on the cloud. You should be enthusiastic about working with cutting-edge technologies in an Agile environment and capable of owning complex components end-to-end, taking an idea through design, testing, and production monitoring.
What will you do:
Responsible for the full lifecycle of our distributed services, focusing on performance, reliability, and innovation: Design & Development
* Architect, design, and implement robust backend services and distributed system components using Go, ensuring high scalability, reliability, and performance.
* Develop microservices and server side infrastructure with clean, maintainable, and testable code.
*  system Ownership: Take full ownership of complex components, moving them from initial concept design through development, testing, CI/CD, and production deployment and monitoring. system Optimization & data
* Optimize service performance, parallel processing, concurrency handling, and resource utilization in high-scale environments.
* Work with a variety of data stores and event-streaming technologies.
* Integrate services with technologies like OpenSearch, Redis, and column store databases as part of the system architecture.
* Implement secure development practices and production ready standards. Collaboration & DevOps
* Work closely with data Engineering, DevOps, and Product teams to deliver high quality, production grade services.
* Develop and deploy applications in Kubernetes environments, leveraging strong DevOps best practices and CI/CD pipelines for smooth, reliable releases.
* Participate actively in code reviews, design discussions, and cross team technical initiatives. Innovation
* Explore new technologies, evaluate tools, and propose improvements to the existing stack and development processes, championing a culture of continuous improvement and technical excellence.