דרושים » תוכנה » Senior Software Engineer - Spark

We're looking for an experienced Software Engineer to join our Data Platform Group. In this key role, you will build the company data platform: cloud-based microservices and data pipelines that process on the order of 1M records/sec at low latency. Because the platform is the foundation other groups build on, your work has a direct impact on our customers and enables engineering, product, and research teams across the organization.
The Lakehouse team owns the data itself - how it is stored, organized, retained, and served. We own our analytical storage layer, the batch processing built on top of it, and the APIs through which customers and the rest of the company consume data. If you enjoy the problems that only appear at petabyte scale - physical data layout, query performance, storage cost, and retention - this is the role.

Responsibilities:
End-to-end ownership of our large-scale analytical storage layer: data modeling, schema and table design, partitioning, retention, and query performance.
Design and develop the batch processing layer over our data lake using Spark on EMR (Java and PySpark).
Build and evolve Java/Spring Boot services that expose our data through well-defined APIs to customers and to consumers across the company.
Own performance and cost: query optimization, file layout and compaction, cluster sizing, and storage efficiency at scale.
Research new technologies in the lakehouse and analytical-storage space and adapt them for use in our product.
Work closely with product, DevOps, and security teams.

We are looking for a Data Engineer to join our team and play a key role in designing, building, and maintaining scalable, cloud-based data pipelines. You will work with AWS services , Airflow and databricks to integrate, process, and analyze large datasets, ensuring data reliability and efficiency.
Your work will directly impact business intelligence, analytics, and data-driven decision-making across the company.
What Youll Do:
ETL & Data Processing: Develop and maintain ETL processes, integrating data from various sources (APIs, databases, external platforms) using Python, SQL, and cloud technologies.
Utilize LangGraph and other frameworks to create state of the art AI agents that integrate various tools and data sources.
Implement integrations that allow agents to take automated actions on behalf of users, streamlining workflows and reducing overhead.
Data Modeling: Design and maintain logical and physical data models to support business needs.
Optimization & Scalability: Improve process efficiency and optimize runtime performance to handle large-scale data workloads.
Collaboration: Work closely with BI analysts and business stakeholders to define data requirements and functional specifications.
Monitoring & Troubleshooting: Ensure data integrity and reliability by proactively monitoring pipelines and resolving issues.

We are looking for a strong, hands-on Data Engineer to join our team and play a key role in building our data infrastructure from the ground up. In this role, you will design and implement scalable data pipelines and platforms, supporting both batch and real-time use cases. You will work closely with analysts and stakeholders to deliver reliable, high-quality data solutions, and take full ownership of data flows - from ingestion to consumption. This is a great opportunity for an executor who enjoys building, moving fast, and making an impact.
What will your job look like?
Design, build, and maintain robust and scalable data pipelines (batch and real-time) end-to-end.
Design and implement scalable, flexible data architectures to support evolving business needs.
Build and manage data platforms, including data lakes and data warehouses.
Integrate multiple data sources (structured and unstructured) into a unified data platform using batch (ETL) and real-time streaming solutions.
Design and implement efficient data models, schemas, and database structures (SQL / NoSQL).
Develop and implement data quality processes to ensure accuracy, consistency, and reliability.
Monitor, optimize, and troubleshoot data infrastructure to meet performance and SLA requirements.

We are seeking talented and passionate Senior Data Engineer to join our Data team. In this pivotal role, you will be instrumental in designing, building, and optimizing the critical data infrastructure that underpins innovative creative intelligence platform. You will tackle complex data challenges, ensuring our systems are robust, scalable, and capable of delivering high-quality data to power our advanced AI models, customer-facing analytics, and internal business intelligence. This is an opportunity to make a significant impact on our product, contribute to a data-driven culture, and help solve fascinating problems at the intersection of data, AI, and marketing technology.

Key Responsibilities
Architect & Develop Data Pipelines: Design, implement, and maintain sophisticated, end-to-end data pipelines for ingesting, processing, validating, and transforming large-scale, diverse datasets.
Manage Data Orchestration: Implement and manage robust workflow orchestration for complex, multi-step data processes, ensuring reliability and visibility.
Advanced Data Transformation & Modeling: Develop and optimize complex data transformations using advanced SQL and other data manipulation techniques. Contribute to the design and implementation of effective data models for analytical and operational use.
Ensure Data Quality & Platform Reliability: Establish and improve processes for data quality assurance, monitoring, alerting, and performance optimization across the data platform. Proactively identify and resolve data integrity and pipeline issues.
Cross-Functional Collaboration: Partner closely with AI engineers, product managers, developers, customer success and other stakeholders to understand data needs, integrate data solutions, and deliver features that provide exceptional value.
Drive Data Platform Excellence: Contribute to the evolution of our data architecture, champion best practices in data engineering (e.g., DataOps principles), and evaluate emerging technologies to enhance platform capabilities, stability, and cost-effectiveness.
Foster a Culture of Learning & Impact: Actively share knowledge, contribute to team growth, and maintain a strong focus on how data engineering efforts translate into tangible product and business outcomes.

Required Senior Data Engineer
Description
We are making the future of Mobility come to life starting today.
We support the worlds largest vehicle fleet operators and transportation providers to optimize existing operations and seamlessly launch new, dynamic business models - driving efficient operations and maximizing utilization.
At the heart of our platform lies the data infrastructure, driving advanced machine learning models and optimization algorithms. As the owner of data pipelines, you'll tackle diverse challenges spanning optimization, prediction, modeling, inference, transportation, and mapping.
As a Senior Data Engineer, you will play a key role in owning and scaling the backend data infrastructure that powers our platform-supporting real-time optimization, advanced analytics, and machine learning applications.
What You'll Do:
Design, implement, and maintain robust, scalable data pipelines for batch and real-time processing using Spark, and other modern tools.
Own the backend data infrastructure, including ingestion, transformation, validation, and orchestration of large-scale datasets.
Leverage Google Cloud Platform (GCP) services to architect and operate scalable, secure, and cost-effective data solutions across the pipeline lifecycle.
Develop and optimize ETL/ELT workflows across multiple environments to support internal applications, analytics, and machine learning workflows.
Build and maintain data marts and data models with a focus on performance, data quality, and long-term maintainability.
Collaborate with cross-functional teams including development teams, product managers, and external stakeholders to understand and translate data requirements into scalable solutions.
Help drive architectural decisions around distributed data processing, pipeline reliability, and scalability.

Required Senior Data Engineer
About the role:
You will specialize in designing and building world class, scalable data architectures, ensuring reliable data flow and integration for groundbreaking biotechnological research. Your expertise in big data tools and pipelines will accelerate our ability to derive actionable insights from complex datasets, driving innovations in improving patients outcome and in delivering life savings treatment solutions.
In this role, you will work closely with data scientists, AI/ML engineers, architect and other cross-functional teams to understand their data needs and requirements. You will also be responsible for ensuring that data is easily accessible and can be used to support data-driven decision making.
Location: Ramat Gan, (Hybrid model)
What will you do?
Design, build, and maintain data pipelines that ingest, transform, and load noisy, heterogeneous biological data at scale - from single-cell sequencing, genomic, and clinical sources - across databases, APIs, and flat files
Enhance our data warehouse system to dynamically support multiple analytics use cases
Implement and productize complex scientific and computational algorithms as scalable, production-grade pipeline components
Drive our data infrastructure toward becoming AI-native, enabling AI/ML systems and agents to reliably query, reason over, and act on our data
Implement data governance policies and procedures to ensure data quality, security, and privacy
Collaborate with ML and AI scientists and other cross-functional teams to understand their data needs and requirements
Drive team capability growth by championing an AI-first SDLC, mentoring data engineers on AI-assisted development practices and tooling
Develop and maintain documentation for data pipelines, processes, and systems
Requirements
We will only consider senior data engineers that have demonstrated strong system design skills combined with an extensive background working with data orchestration, data warehousing and ETL tools.

We are seeking a highly skilled and analytical Senior Data Engineer to join our Data team. In this role, you will design and implement robust data pipelines, while uniquely bridging the gap between engineering and analytics by actively analyzing data to extract actionable insights. You will play a crucial part in architecting our data foundations to support everything from business intelligence to advanced machine learning and agentic AI pipelines.
As a Senior Data Engineer, you will collaborate closely with engineering teams, product managers, and stakeholders across the organization. You will not only build the infrastructure utilizing modern data stack tools but also act as a data analyst when needed, ensuring our systems are fully equipped to operate within and support a cutting-edge agentic AI environment.
Responsibilities
Design, build, and maintain highly scalable ELT/ETL data pipelines.
Architect and manage modern cloud data warehousing solutions.
Develop, maintain, and monitor Python services responsible for robust data collection and ingestion.
Perform hands-on data analysis to interpret complex datasets, identify trends, and deliver business insights, acting in a dual capacity as a Data Analyst.
Develop and optimize data infrastructure specifically designed to support autonomous agentic workflows and LLM integrations.
Collaborate with engineers and analysts to troubleshoot data issues, enforce quality SLAs, and define data requirements.
Document data architecture, flow, and analytics standards for internal team alignment.
Build and maintain dashboards and reports to communicate analytical findings and data health to the organization.
Maintain Kafka consumer applications that process high-volume event streams in real-time, ensuring reliable ingestion into cloud databases.

our company seeks an experienced Senior Staff Data Engineer to join our AI and Data Solutions unit and champion an AI-first strategy. You will spearhead the development of Data Lakehouse processes, model core product domains, and define data engineering best practices across the organization. Serving as a technical project lead, you will provide strong direction and collaborate with Data Science, Analytics, R&D, and IT teams to elevate onboarding workflows. This role demands a proactive approach to optimizing system capabilities and driving cross-functional data initiatives.
Key Responsibilities
Own the design, implementation, and optimization of scalable Data Lakehouse and Data Warehouse architectures across diverse cloud ecosystems.
Spearhead the development of robust ETL/ELT pipelines and data ingestion processes utilizing SQL, Python, PySpark, and DBT.
Apply advanced data modelling techniques to structure complex data assets into optimized facts, dimensions, and partitions.
Partner with cross-functional leadership across data science, analytics, R&D, and IT to identify requirements and improve workflows.
Orchestrate complex data workflows combining tools like Airflow and DBT with advanced Gen-AI development workflows using CloudCode.
Define, document, and popularize data engineering best practices, design patterns, and data quality standards across the unit.
Evaluate and implement modern data technologies spanning vendor-specific and open-source landscapes including GCP, AWS, and Apache Iceberg.
Provide technical direction and hands-on guidance to ensure high-performance execution, reliable pipeline delivery, and proactive problem-solving.

we are a dynamic technology firm dedicated to building exceptional products using modern technologies for the world's most admired companies. We pride ourselves on innovation, collaboration, and delivering excellence in every project.

Job Description
Responsibilities:

Design and maintain scalable data pipelines with Spark Structured Streaming.
Implement Lakehouse architecture with Apache Iceberg.
Develop ETL processes for data transformation.
Ensure data integrity, quality, and governance.
Collaborate with stakeholders and IT teams for seamless solution integration.
Optimize data processing workflows and performance.

As a Senior Data Engineer, youll collaborate with top-notch engineers and data scientists to elevate our platform to the next level and deliver exceptional user experiences. Your primary focus will be on the data engineering aspects-ensuring the seamless flow of high-quality, relevant data to train and optimize content models, including GenAI foundation models, supervised fine-tuning, and more.

Youll work closely with teams across the company to ensure the availability of high-quality data from ML platforms, powering decisions across all departments. With access to petabytes of data through MySQL, Snowflake, Cassandra, S3, and other platforms, your challenge will be to ensure that this data is applied even more effectively to support business decisions, train and monitor ML models and improve our products.



Key Job Responsibilities and Duties:

Rapidly developing next-generation scalable, flexible, and high-performance data pipelines.

Dealing with massive textual sources to train GenAI foundation models.

Solving issues with data and data pipelines, prioritizing based on customer impact.

End-to-end ownership of data quality in our core datasets and data pipelines.

Experimenting with new tools and technologies to meet business requirements regarding performance, scaling, and data quality.

Providing tools that improve Data Quality company-wide, specifically for ML scientists.

Providing self-organizing tools that help the analytics community discover data, assess quality, explore usage, and find peers with relevant expertise.

Acting as an intermediary for problems, with both technical and non-technical audiences.

Promote and drive impactful and innovative engineering solutions

Technical, behavioral and interpersonal competence advancement via on-the-job opportunities, experimental projects, hackathons, conferences, and active community participation

Collaborate with multidisciplinary teams: Collaborate with product managers, data scientists, and analysts to understand business requirements and translate them into machine learning solutions. Provide technical guidance and mentorship to junior team members.