דרושים » תוכנה » Senior Data Engineer

We are now looking for a Data Engineer to join our team and play a key role in building and optimizing large-scale Big Data systems in production environments.

Key Responsibilities:
Design, implement, and maintain Big Data pipelines in production.
Work extensively with Apache Spark (2.x and above), focusing on complex joins, shuffle optimization, and performance improvements at scale.
Integrate Spark with relational databases, NoSQL systems, cloud storage, and streaming platforms.
Contribute to system architecture and ensure scalability, reliability, and efficiency in data processing workflows.

Required Senior Data Engineer
As part of the Audiences team, youll work on highly scalable systems, tackle real-time challenges, and contribute to a platform that directly impacts the success of our customers campaigns.
Our team mixes disciplines from back-end, data, AI and front-end engineers and were looking for a great back-end data oriented developer that is passionate about what they do, who want to dive into the worlds of big data, microservices and production management at our scale, to help us reach our vision and goals
What youll do
Solve diverse, complex problems at a large scale
Collaborate with Product Managers, data scientists and Software Engineers to deliver innovative, user-facing products.
Independent - cradle to grave, own a feature from scratch to production
Analyze and improve performance, scalability and stability of our products, environments and tools
Continuously learn and evaluate new technologies in the everlasting effort to perfect our products and adapt to the ever-growing scale.

Are you a software engineer passionate about building cutting-edge systems?enior
Are you excited about shaping the future of security through Big Data and AI?
Who We Are?
We're part of the Cloud Security Intelligence group, owning one of the largest Big Data environments in Israel.
Partner with the best
Our team builds and operates intelligence-driven products that enable customers to detect and respond to security threats at scale.
As a Senior Software & AI Software Engineer, you'll play a key role in
Designing, developing, and deploying cloud-native, high-scale Big Data systems, Cloud (Azure/GCP).
Building and optimizing data pipelines (Spark, Delta Lake, Airflow, Databricks).Integrating AI/ML models into production environments to deliver real-time insights.
Leading AI-driven initiatives, from PoCs to customer-facing features.
Collaborating across product, UX, data science, and engineering to shape the next generation of security tools.
Leveraging technologies such as Kubernetes, Kafka, ELK, Prometheus, Grafana, and more.

We are seeking a Big Data Engineer with at least 3 years of experience. A strong understanding of data structure is critical.
You will design, develop and maintain data systems, optimize pipelines, and ensure data integrity.
Data Engineers shape and improve leading mobile analytics solutions while using cutting edge technologies such as Scala, Spark, Hadoop, Astronomer, BigQuery, and many more.
What youll do
Own and develop our products and features end-to-end - from design to production
Collaborate with Product Managers and Software Engineers to deliver complex products.
Develop and maintain scalable data pipelines.
Implement ETL processes and data integration.
Optimize data performance and ensure security.

Does building the next generation of AI/ML platforms excite you?
Do Big Data challenges and open-source innovation speak your language?
Join our Aegis team
Aegis is an end-to-end Big Data AI/ML platform built on top of Linode. It bridges the gap between research and production, enabling teams to innovate faster and deliver ML products more efficiently. We leverage open-source tools and AI accelerators to avoid reinventing the wheel integrating across platforms where it matters most.
Make a difference in your own way
You'll join our growing Engineering group, working hands-on to shape a powerful, flexible ML platform that supports real-world, large-scale data and AI workflows. You'll be a key contributor to an environment that spans Kubernetes, Spark, MLFlow, JupyterHub, and PyTorch and that pushes the boundaries of performance and collaboration.
In this role you will be:
Designing and implementing scalable data and ML pipelines using Spark, MLFlow, and Kubernetes
Building platform components in Python or Scala to connect research and production environments
Integrating with orchestration tools such as Argo Workflows (or equivalents)
Supporting hybrid data environments across S3/ADLS and cloud/on-prem compute
Collaborating with researchers and MLOps engineers to deliver reproducible and high-performance workflows.

We are looking for a skilled data engineer to join a cutting-edge project supporting a military technology unit. This role offers a unique opportunity to work on innovative cloud-based data infrastructure within a secure and challenging environment.

Responsibilities:
Data Infrastructure Development: Design, build, and manage advanced data infrastructure in Google Cloud Platform (GCP) to support the operational and analytical needs of the units clients.
Data Pipeline Engineering: Develop robust and scalable data pipelines in GCP, perform integrations with diverse data sources, and enable AI/ML workflows using tools such as Dataflow, Pub/Sub, and BigQuery.
ETL & DWH Automation: Improve, optimize, and automate ETL processes and data warehouse operations in the cloud.
Cross-Organizational Tech Support: Participate in high-impact, cross-organizational technology projects in a dynamic and innovative technical environment.

Required Senior Data-Oriented Backend Engineer
As part of the Audiences team, youll work on highly scalable systems, tackle real-time challenges, and contribute to a platform that directly impacts the success of our customers campaigns.
Our team mixes disciplines from back-end, data, AI and front-end engineers and were looking for a great back-end data oriented developer that is passionate about what they do, who want to dive into the worlds of big data, microservices and production management at our scale, to help us reach our vision and goals
What youll do
Solve diverse, complex problems at a large scale
Collaborate with Product Managers, data scientists and Software Engineers to deliver innovative, user-facing products.
Independent - cradle to grave, own a feature from scratch to production
Analyze and improve performance, scalability and stability of our products, environments and tools
Continuously learn and evaluate new technologies in the everlasting effort to perfect our products and adapt to the ever-growing scale.

At Infinidat, we help enterprises and service providers empower their data-driven competitive advantage at scale. We are a leading provider of enterprise-class storage solutions. The company’s software-focused architecture delivers sub-millisecond latency, full availability, and scalability with a significantly lower total cost of ownership than competing storage technologies. We’re looking for passionate and bright individual who wish to take part in the design, development and all technical-related aspects of INFINIDAT’s S3 Protocol Development team. As a Software Engineer in the S3 Protocol Development team, you will design and implement core features of an S3-compatible object storage system. You will work on high-performance data paths, distributed systems logic, and protocol compliance to ensure seamless integration with high performance applications and data protection applications. Your work will directly.

Responsibilities:


* Design, develop, and optimize core S3 protocol components in a large-scale distributed storage system.
* Ensure compliance with AWS S3 API standards and extend functionality with advanced features.
* Collaborate with cross-functional teams on scalability and performance optimizations.
* Troubleshoot and resolve complex issues in development and test environments.
* Participate in code reviews, design discussions, and architectural decision-making.
* Continuously explore and evaluate new technologies and approaches to enhance protocol efficiency and system reliability.

Required Data Engineer
Description
Join our core platform engineering team, developing our AI-powered automotive data management platform.
We are developing the next generation data-driven products for the Automotive industry, focusing on cybersecurity (XDR) and vehicle quality. Our products monitor and secure millions of vehicles worldwide and help automakers leverage connected vehicle data to deliver cyber resilience, safety, customer satisfaction and increase brand loyalty.
Our Data Engineering & Data Science Group leads the development of our Iceberg-based data platform, including data lake, query engine, and ML-Ops tools, serving as a solid AI-ready foundation for all our products.
At the core of our Engineering Team, you will build and operate scalable, production-grade customer-facing data and ML platform components, focusing on reliability and performance.
Technological background and focus: Iceberg, Trino, Prefect, GitHub Actions, Kubernetes, JupyterHub, MLflow, dbt
This role is full-time and is Herzliya, Israel based.
Responsibilities
Design, build, and maintain scalable data pipelines to ingest and transform batch data on our data lake, enabling analytics and ML with strong data quality, governance, observability, and CI/CD.
Build and expand our foundational data infrastructure, including our data lake, analytics engine, and batch processing frameworks.
Create robust infrastructure to enable automated pipelines that will ingest and process data into our analytical platforms, leveraging open-source, cloud-agnostic frameworks and toolsets.
Develop and maintain our data lake layouts and architectures for efficient data access and advanced analytics.
Build our ML platform and automate the ML lifecycle.
Drive business-wide ML projects from an engineering perspective.
Develop and manage orchestration tools, governance tools, data discovery tools, and more.
Work with other team members of the engineering group, including data architects, data analysts, and data scientists, to provide solutions using a use case-based approach that drives the construction of technical data flows.

We are seeking a seasoned and strategic Head of BI to lead our Business Intelligence group.
In this pivotal role, you will be responsible for overseeing the full BI stackfrom data infrastructure and pipelines to analytics and reportingwhile managing a team of data engineers and analysts.
You will drive the design and execution of our data strategy, ensure delivery of trusted insights across the organization, and lead the team responsible for building and maintaining our data platform and analytics capabilities. This is a hands-on leadership role requiring a balance of technical depth, managerial experience, and strategic thinking.Responsibilities
What Youll Be Doing:
Lead, mentor, and scale a cross-functional team of BI Analysts and Data Engineers.
Own and evolve our data warehouse architecture, ensuring integrity, scalability, and performance (Snowflake preferred).
Oversee the design and development of ETL/ELT pipelines and semantic data models using tools such as dbt, Python, and Airflow.
Collaborate with senior stakeholders to align business goals with data priorities, translating requirements into scalable solutions.
Define and implement best practices across the entire BI lifecycle: data ingestion, transformation, modeling, visualization, and governance.
Deliver robust, actionable dashboards and insights that drive key business decisions and KPIs.