דרושים » ניהול ביניים » Big Data Infrastructure Engineer

We are seeking talented and passionate Senior Data Engineer to join our Data team. In this pivotal role, you will be instrumental in designing, building, and optimizing the critical data infrastructure that underpins innovative creative intelligence platform. You will tackle complex data challenges, ensuring our systems are robust, scalable, and capable of delivering high-quality data to power our advanced AI models, customer-facing analytics, and internal business intelligence. This is an opportunity to make a significant impact on our product, contribute to a data-driven culture, and help solve fascinating problems at the intersection of data, AI, and marketing technology.

Key Responsibilities
Architect & Develop Data Pipelines: Design, implement, and maintain sophisticated, end-to-end data pipelines for ingesting, processing, validating, and transforming large-scale, diverse datasets.
Manage Data Orchestration: Implement and manage robust workflow orchestration for complex, multi-step data processes, ensuring reliability and visibility.
Advanced Data Transformation & Modeling: Develop and optimize complex data transformations using advanced SQL and other data manipulation techniques. Contribute to the design and implementation of effective data models for analytical and operational use.
Ensure Data Quality & Platform Reliability: Establish and improve processes for data quality assurance, monitoring, alerting, and performance optimization across the data platform. Proactively identify and resolve data integrity and pipeline issues.
Cross-Functional Collaboration: Partner closely with AI engineers, product managers, developers, customer success and other stakeholders to understand data needs, integrate data solutions, and deliver features that provide exceptional value.
Drive Data Platform Excellence: Contribute to the evolution of our data architecture, champion best practices in data engineering (e.g., DataOps principles), and evaluate emerging technologies to enhance platform capabilities, stability, and cost-effectiveness.
Foster a Culture of Learning & Impact: Actively share knowledge, contribute to team growth, and maintain a strong focus on how data engineering efforts translate into tangible product and business outcomes.

Were looking for a Big Data Engineer to develop and integrate systems that retrieve, process and analyze data from around the digital world, generating customer-facing data. This role will report to our Team Manager, R&D.
Why is this role so important?
A data-focused company, and data is the heart of our business.
As a big data engineer developer, you will work at the very core of the company, designing and implementing complex high scale systems to retrieve and analyze data from millions of digital users.
Your role as a big data engineer will give you the opportunity to use the most cutting-edge technologies and best practices to solve complex technical problems while demonstrating technical leadership.
So, what will you be doing all day?
Your role as part of the R&D team means your daily responsibilities may include:
Design and implement complex high scale systems using a large variety of technologies.
You will work in a data research team alongside other data engineers, data scientists and data analysts. Together you will tackle complex data challenges and bring new solutions and algorithms to production.
Contribute and improve the existing infrastructure of code and data pipelines, constantly exploring new technologies and eliminating bottlenecks.
You will experiment with various technologies in the domain of Machine Learning and big data processing.
You will work on a monitoring infrastructure for our data pipelines to ensure smooth and reliable data ingestion and calculation.

We are seeking an experienced Solutions Data Engineer who possess both technical depth and strong interpersonal skills to partner with internal and external teams to develop scalable, flexible, and cutting-edge solutions. Solutions Engineers collaborate with operations and business development to help craft solutions to meet customer business problems.
A Solutions Engineer works to balance various aspects of the project, from safety to design. Additionally, a Solutions Engineer researches advanced technology regarding best practices in the field and seek to find cost-effective solutions.
Job Description:
Were looking for a Solutions Engineer with deep experience in Big Data technologies, real-time data pipelines, and scalable infrastructure-someone whos been delivering critical systems under pressure, and knows what it takes to bring complex data architectures to life. This isnt just about checking boxes on tech stacks-its about solving real-world data problems, collaborating with smart people, and building robust, future-proof solutions.
In this role, youll partner closely with engineering, product, and customers to design and deliver high-impact systems that move, transform, and serve data at scale. Youll help customers architect pipelines that are not only performant and cost-efficient but also easy to operate and evolve.
We want someone whos comfortable switching hats between low-level debugging, high-level architecture, and communicating clearly with stakeholders of all technical levels.
Key Responsibilities:
Build distributed data pipelines using technologies like Kafka, Spark (batch & streaming), Python, Trino, Airflow, and S3-compatible data lakes-designed for scale, modularity, and seamless integration across real-time and batch workloads.
Design, deploy, and troubleshoot hybrid cloud/on-prem environments using Terraform, Docker, Kubernetes, and CI/CD automation tools.
Implement event-driven and serverless workflows with precise control over latency, throughput, and fault tolerance trade-offs.
Create technical guides, architecture docs, and demo pipelines to support onboarding, evangelize best practices, and accelerate adoption across engineering, product, and customer-facing teams.
Integrate data validation, observability tools, and governance directly into the pipeline lifecycle.
Own end-to-end platform lifecycle: ingestion → transformation → storage (Parquet/ORC on S3) → compute layer (Trino/Spark).
Benchmark and tune storage backends (S3/NFS/SMB) and compute layers for throughput, latency, and scalability using production datasets.
Work cross-functionally with R&D to push performance limits across interactive, streaming, and ML-ready analytics workloads.
Operate and debug object store-backed data lake infrastructure, enabling schema-on-read access, high-throughput ingestion, advanced searching strategies, and performance tuning for large-scale workloads.

As a Senior Software Engineer your role will be to:

Play a key role in the design and architecture of cutting-edge, innovative data systems.
Develop and maintain a multi-tenant environment for our advanced measurement solutions.
Build and manage cloud-based, distributed computing environments to support scalable, high-performance operations.
Design and implement robust ETL workflows, integrating multiple data sources for cleaning, transformation, aggregation, and querying.
Troubleshoot and resolve complex issues related to data integration, storage, and retrieval.
Architect and build big data solutions capable of managing and automating the processing of tens of terabytes.
Optimize database performance and streamline query execution in large-scale environments.
Collaborate closely with Data Scientists to align technical solutions with data-driven objectives.

As a Senior Software Engineer your role will be to:

Play a key role in the design and architecture of cutting-edge, innovative data systems.
Develop and maintain a multi-tenant environment for our advanced measurement solutions.
Build and manage cloud-based, distributed computing environments to support scalable, high-performance operations.
Design and implement robust ETL workflows, integrating multiple data sources for cleaning, transformation, aggregation, and querying.
Troubleshoot and resolve complex issues related to data integration, storage, and retrieval.
Architect and build big data solutions capable of managing and automating the processing of tens of terabytes.
Optimize database performance and streamline query execution in large-scale environments.
Collaborate closely with Data Scientists to align technical solutions with data-driven objectives.

We are looking for a Data Engineer to join our team and play a key role in designing, building, and maintaining scalable, cloud-based data pipelines. You will work with AWS services , Airflow and databricks to integrate, process, and analyze large datasets, ensuring data reliability and efficiency.
Your work will directly impact business intelligence, analytics, and data-driven decision-making across the company.
What Youll Do:
ETL & Data Processing: Develop and maintain ETL processes, integrating data from various sources (APIs, databases, external platforms) using Python, SQL, and cloud technologies.
Utilize LangGraph and other frameworks to create state of the art AI agents that integrate various tools and data sources.
Implement integrations that allow agents to take automated actions on behalf of users, streamlining workflows and reducing overhead.
Data Modeling: Design and maintain logical and physical data models to support business needs.
Optimization & Scalability: Improve process efficiency and optimize runtime performance to handle large-scale data workloads.
Collaboration: Work closely with BI analysts and business stakeholders to define data requirements and functional specifications.
Monitoring & Troubleshooting: Ensure data integrity and reliability by proactively monitoring pipelines and resolving issues.

We are looking for an experienced Senior Data Engineer to join our Data team. In this role, you will lead and strengthen our Data Team, drive innovation, and ensure the robustness of our data/analytics platforms.
A day in the life and how youll make an impact:
Design and develop high-performance Data Pipelines and ETL processes to support diverse business needs.
Work closely with Product, Business Intelligence, Sales and other teams to integrate data solutions, ensuring seamless alignment and collaboration across functions.
Continuously improve our data analytics platforms, optimizing system performance and ensuring robust, reliable data infrastructure.
Oversee the entire data lifecycle, from infrastructure setup and data acquisition to detailed analysis and automated reporting, driving business growth through data-driven insights.
Implement robust data quality checks, monitoring mechanisms, and data governance policies to maintain data integrity and security, troubleshooting and resolving any data-related issues efficiently.

Were looking for a Senior Data Engineer in the Data Collection Ingest team to contribute to the design and coding of our data ingestion services & pipelines. This role involves working as part of a team that handles millions of requests per minute across multiple servers, and is responsible for a wide-range of data pipelines, processing billions of events each day.
Why is this role so important?
As the most trusted platform for measuring online behavior, millions of people rely on our insights daily as the ground truth for their knowledge of the digital world. Producing these insights requires large scale raw data to be ingested reliably in high scale to provide stable signals for analysis. As a Data Engineer you will have the opportunity to perform hands-on work and own our raw data ingestion pipeline end-to-end. Your work will have a direct impact on the quality and reliability of our data and the insights that our products are delivering to our customers.
So, what will you be doing all day?
Your role as Data Engineer of the Ingest Data Collection team means your daily responsibilities may include:
Design, code and manage end-to-end our data ingestion pipelines, both online and offline.
Take charge of developing & maintaining modern data infrastructure, while implementing best practices for building data pipelines.
Be responsible for high-scale ingestion services, solving challenges of availability, reliability, and scalability.
Run the production environment by monitoring availability and taking a holistic view of system health and data quality.
Own data infrastructure features from design to production using industry best practices with focus on quality and delivery.
Lead design & decision-making processes of the team.
Solve diverse complex problems of scale, performance and business logic.
Collaborate with product managers and other team leaders to plan, nurture, and implement an efficient and effective development process.
Continuously learn and evaluate new technologies in the everlasting effort to perfect our products
Perform code reviews, evaluate implementations, and provide feedback about potential improvements.
Improve your skills, learn from and mentor top-notch engineers and enrich other team members.
Have lots of fun!

Required Senior Principal Data Scientist - LLM Agents (Cortex)
Job Summary:
Are you passionate about harnessing cutting-edge data science techniques to protect the world from cyber threats? Do you thrive in the dynamic world of cybersecurity? If so, we invite you to join our innovative and forward-thinking team, where you'll have the opportunity to shape the future of cybersecurity and impact millions of customers.
As a Senior Principal Data Scientist, you will be a technical leader and hands-on architect, bridging the worlds of advanced data science, robust software engineering, and next-generation AI. You will bring a deep, diverse background in classical machine learning algorithms alongside cutting-edge LLM agent architectures to build resilient, production-grade defense techniques against cyber-villains. Operating as a premier technical authority, you will drive the vision, design, and large-scale development of our groundbreaking, Agentix LLM-based security solutions that block attacks even before they begin.
Key Responsibilities:
Serve as a hands-on technical compass for a diverse, elite product-oriented research team, pioneering state-of-the-art technologies and setting the engineering standards for AI/ML excellence across the organization.
Design, implement, and optimize robust Agentic LLM solutions and multi-agent workflows engineered to autonomously counter modern, sophisticated cyber threats.
Oversee and actively contribute to the entire end-to-end lifecycle, moving seamlessly from ambiguous research concepts and POCs to high-throughput, low-latency production deployments.
Build and enforce comprehensive frameworks for continuous real-time monitoring, evaluation, and guardrails to track model drift, latency, and agent behavior in production environments.
Decompose highly complex, multi-layered algorithmic problems into actionable technical roadmaps and strategic insights for both deeply technical engineers and executive business stakeholders

Are you passionate about harnessing cutting-edge data science techniques to protect the world from cyber threats? Do you thrive in the dynamic world of cybersecurity? If so, we invite you to join our innovative and forward-thinking team at Palo Alto Networks, where you'll have the opportunity to shape the future of cybersecurity and impact millions of customers.
As a Senior Principal Data Scientist, you will be a technical leader and hands-on architect, bridging the worlds of advanced data science, robust software engineering, and next-generation AI. You will bring a deep, diverse background in classical machine learning algorithms alongside cutting-edge LLM agent architectures to build resilient, production-grade defense techniques against cyber-villains. Operating as a premier technical authority, you will drive the vision, design, and large-scale development of our groundbreaking, Agentix LLM-based security solutions that block attacks even before they begin.
Key Responsibilities
Serve as a hands-on technical compass for a diverse, elite product-oriented research team, pioneering state-of-the-art technologies and setting the engineering standards for AI/ML excellence across the organization.
Design, implement, and optimize robust Agentic LLM solutions and multi-agent workflows engineered to autonomously counter modern, sophisticated cyber threats.
Oversee and actively contribute to the entire end-to-end lifecycle, moving seamlessly from ambiguous research concepts and POCs to high-throughput, low-latency production deployments.
Build and enforce comprehensive frameworks for continuous real-time monitoring, evaluation, and guardrails to track model drift, latency, and agent behavior in production environments.
Decompose highly complex, multi-layered algorithmic problems into actionable technical roadmaps and strategic insights for both deeply technical engineers and executive business stakeholders.