דרושים » דאטה » Core Data - Big Data Engineer

Were looking for a Senior Data Engineer in the Data Collection Ingest team to contribute to the design and coding of our data ingestion services & pipelines. This role involves working as part of a team that handles millions of requests per minute across multiple servers, and is responsible for a wide-range of data pipelines, processing billions of events each day.
Why is this role so important?
As the most trusted platform for measuring online behavior, millions of people rely on our insights daily as the ground truth for their knowledge of the digital world. Producing these insights requires large scale raw data to be ingested reliably in high scale to provide stable signals for analysis. As a Data Engineer you will have the opportunity to perform hands-on work and own our raw data ingestion pipeline end-to-end. Your work will have a direct impact on the quality and reliability of our data and the insights that our products are delivering to our customers.
So, what will you be doing all day?
Your role as Data Engineer of the Ingest Data Collection team means your daily responsibilities may include:
Design, code and manage end-to-end our data ingestion pipelines, both online and offline.
Take charge of developing & maintaining modern data infrastructure, while implementing best practices for building data pipelines.
Be responsible for high-scale ingestion services, solving challenges of availability, reliability, and scalability.
Run the production environment by monitoring availability and taking a holistic view of system health and data quality.
Own data infrastructure features from design to production using industry best practices with focus on quality and delivery.
Lead design & decision-making processes of the team.
Solve diverse complex problems of scale, performance and business logic.
Collaborate with product managers and other team leaders to plan, nurture, and implement an efficient and effective development process.
Continuously learn and evaluate new technologies in the everlasting effort to perfect our products
Perform code reviews, evaluate implementations, and provide feedback about potential improvements.
Improve your skills, learn from and mentor top-notch engineers and enrich other team members.
Have lots of fun!

We are seeking talented and passionate Senior Data Engineer to join our Data team. In this pivotal role, you will be instrumental in designing, building, and optimizing the critical data infrastructure that underpins innovative creative intelligence platform. You will tackle complex data challenges, ensuring our systems are robust, scalable, and capable of delivering high-quality data to power our advanced AI models, customer-facing analytics, and internal business intelligence. This is an opportunity to make a significant impact on our product, contribute to a data-driven culture, and help solve fascinating problems at the intersection of data, AI, and marketing technology.

Key Responsibilities
Architect & Develop Data Pipelines: Design, implement, and maintain sophisticated, end-to-end data pipelines for ingesting, processing, validating, and transforming large-scale, diverse datasets.
Manage Data Orchestration: Implement and manage robust workflow orchestration for complex, multi-step data processes, ensuring reliability and visibility.
Advanced Data Transformation & Modeling: Develop and optimize complex data transformations using advanced SQL and other data manipulation techniques. Contribute to the design and implementation of effective data models for analytical and operational use.
Ensure Data Quality & Platform Reliability: Establish and improve processes for data quality assurance, monitoring, alerting, and performance optimization across the data platform. Proactively identify and resolve data integrity and pipeline issues.
Cross-Functional Collaboration: Partner closely with AI engineers, product managers, developers, customer success and other stakeholders to understand data needs, integrate data solutions, and deliver features that provide exceptional value.
Drive Data Platform Excellence: Contribute to the evolution of our data architecture, champion best practices in data engineering (e.g., DataOps principles), and evaluate emerging technologies to enhance platform capabilities, stability, and cost-effectiveness.
Foster a Culture of Learning & Impact: Actively share knowledge, contribute to team growth, and maintain a strong focus on how data engineering efforts translate into tangible product and business outcomes.

As a Data Engineer, youll collaborate with top-notch engineers and data scientists to elevate our platform to the next level and deliver exceptional user experiences. Your primary focus will be on the data engineering aspects-ensuring the seamless flow of high-quality, relevant data to train and optimize content models, including GenAI foundation models, supervised fine-tuning, and more.

Youll work closely with teams across the company to ensure the availability of high-quality data from ML platforms, powering decisions across all departments. With access to petabytes of data through MySQL, Snowflake, Cassandra, S3, and other platforms, your challenge will be to ensure that this data is applied even more effectively to support business decisions, train and monitor ML models and improve our products.


Key Job Responsibilities and Duties:

Rapidly developing next-generation scalable, flexible, and high-performance data pipelines.

Dealing with massive textual sources to train GenAI foundation models.

Solving issues with data and data pipelines, prioritizing based on customer impact.

End-to-end ownership of data quality in our core datasets and data pipelines.

Experimenting with new tools and technologies to meet business requirements regarding performance, scaling, and data quality.

Providing tools that improve Data Quality company-wide, specifically for ML scientists.

Providing self-organizing tools that help the analytics community discover data, assess quality, explore usage, and find peers with relevant expertise.

Acting as an intermediary for problems, with both technical and non-technical audiences.

Promote and drive impactful and innovative engineering solutions

Technical, behavioral and interpersonal competence advancement via on-the-job opportunities, experimental projects, hackathons, conferences, and active community participation

Collaborate with multidisciplinary teams: Collaborate with product managers, data scientists, and analysts to understand business requirements and translate them into machine learning solutions. Provide technical guidance and mentorship to junior team members.

As a Senior Data Engineer, youll collaborate with top-notch engineers and data scientists to elevate our platform to the next level and deliver exceptional user experiences. Your primary focus will be on the data engineering aspects-ensuring the seamless flow of high-quality, relevant data to train and optimize content models, including GenAI foundation models, supervised fine-tuning, and more.

Youll work closely with teams across the company to ensure the availability of high-quality data from ML platforms, powering decisions across all departments. With access to petabytes of data through MySQL, Snowflake, Cassandra, S3, and other platforms, your challenge will be to ensure that this data is applied even more effectively to support business decisions, train and monitor ML models and improve our products.



Key Job Responsibilities and Duties:

Rapidly developing next-generation scalable, flexible, and high-performance data pipelines.

Dealing with massive textual sources to train GenAI foundation models.

Solving issues with data and data pipelines, prioritizing based on customer impact.

End-to-end ownership of data quality in our core datasets and data pipelines.

Experimenting with new tools and technologies to meet business requirements regarding performance, scaling, and data quality.

Providing tools that improve Data Quality company-wide, specifically for ML scientists.

Providing self-organizing tools that help the analytics community discover data, assess quality, explore usage, and find peers with relevant expertise.

Acting as an intermediary for problems, with both technical and non-technical audiences.

Promote and drive impactful and innovative engineering solutions

Technical, behavioral and interpersonal competence advancement via on-the-job opportunities, experimental projects, hackathons, conferences, and active community participation

Collaborate with multidisciplinary teams: Collaborate with product managers, data scientists, and analysts to understand business requirements and translate them into machine learning solutions. Provide technical guidance and mentorship to junior team members.

We are looking for a Data Engineer to help build and scale the data platform behind our search quality, ML pipelines, product analytics, and business operations.
In this role, you will contribute across the full data lifecycle: ingesting data from production systems, designing and evolving our data warehouse, building batch and streaming pipelines, and making high-quality datasets available to researchers, engineers, analysts, and product teams across the company.
The platform spans tens of terabytes and ingests data from tens of proprietary and third-party sources - including our search engine and its components, CRM, billing, identity, and product analytics across multi-region production environments. Around 100 internal users rely on it daily.
You will work closely with engineers and stakeholders across the company, contribute to architectural and modeling decisions, and help improve the reliability, usability, and scalability of the data platform as it grows.

In this position, your responsibility will be to:
Contribute to the design, development, and operation of Tavily's data platform - from real-time ingestion through data warehouse medallion layers to consumer-facing datasets and dashboards.
Build and maintain reliable batch and streaming pipelines that ingest data from production services and external systems.
Design and evolve scalable, analytics-ready data models in the data warehouse.
Work closely with engineers across the company to ensure data produced by production systems is reliable, well-structured, and usable downstream.
Improve observability across the data platform, including data quality checks, freshness monitoring, lineage, schema evolution, and cost controls.
Partner with researchers, engineers, analysts, finance, and product managers to deliver trustworthy datasets for product, search quality, ML, and GTM analytics.
Contribute to defining the objects, entities, and relationships that represent Tavily's search domain - including agent inputs, URLs, chunks, agent sessions, crawls, and the connections between them - and translate them into clean, queryable data models.
Improve engineering practices around testing, documentation, deployment, and incident response.
Investigate and resolve production data issues, including broken pipelines, corrupted datasets, schema changes, and large-scale backfills.
Contribute to technical standards and best practices for data engineering across the company.
Help maintain high standards of data quality, integrity, security, and governance across environments.

As our Senior Data Engineer you will handle Big Data in real time, and become the owner and gatekeeper of all data and data flows within the company. You will bring data ingenuity and technological excellence while gaining a deep business understanding.

This is an amazing opportunity to join a multi-disciplinary A-team while working in a fast-paced, modern-cloud, data-oriented environment.

Implement robust, reliable, and scalable data pipelines and data architecture.
Own and develop DWH (petabytes of data!).
Own the entire data development process, including business knowledge, methodology, quality assurance, and monitoring.
Collaborate with cross-functional teams to define, design, and ship new features.
Continuously discover, evaluate, and implement new technologies to maximize development efficiency.
Develop tailor-made solutions as part of our data pipelines.
Lead complex Big Data projects and build data platforms from scratch.
Work on high-scale, real-time, real-world, business-critical data stores and data endpoints.
Implement data profiling to identify anomalies and maintain data integrity.
Work in a results-driven, high-paced, rewarding environment.

We are looking for a Data Engineer to join our team and play a key role in designing, building, and maintaining scalable, cloud-based data pipelines. You will work with AWS services , Airflow and databricks to integrate, process, and analyze large datasets, ensuring data reliability and efficiency.
Your work will directly impact business intelligence, analytics, and data-driven decision-making across the company.
What Youll Do:
ETL & Data Processing: Develop and maintain ETL processes, integrating data from various sources (APIs, databases, external platforms) using Python, SQL, and cloud technologies.
Utilize LangGraph and other frameworks to create state of the art AI agents that integrate various tools and data sources.
Implement integrations that allow agents to take automated actions on behalf of users, streamlining workflows and reducing overhead.
Data Modeling: Design and maintain logical and physical data models to support business needs.
Optimization & Scalability: Improve process efficiency and optimize runtime performance to handle large-scale data workloads.
Collaboration: Work closely with BI analysts and business stakeholders to define data requirements and functional specifications.
Monitoring & Troubleshooting: Ensure data integrity and reliability by proactively monitoring pipelines and resolving issues.

We are looking for a Senior Data Engineer to build and scale the data foundation behind platform and products. You will own complex data end-to-end - from ingestion and transformation through modeling, quality, observability, and production delivery.
This is a hands-on senior IC role for a strong builder who can solve difficult data problems independently, set a high technical bar, and collaborate closely with engineering, DS, and product. You will turn large, fragmented datasets into reliable, reusable capabilities that power every product.
Responsibilities
Build and own scalable data pipelines- Design, implement, and operate robust pipelines for high-volume structured and unstructured data, with validation, monitoring, lineage, and recovery built in.
Scale the platform for growth- A key near-term initiative is re-architecting the system to support a significantly larger customer base. You will own performance and cost-efficiency across pipelines and services, keeping reliability and operating costs under control as the platform scales.
Build across the stack- This is not a pipelines-only role. You will also write backend services and some frontend, including the internal backoffice the team runs on. We hire builders, not narrow specialists.
Own the core data tables- Own schema design and evolution, data contracts, and the modeling standards the team follows - naming, shared dimensions, normalization, documentation. Be accountable when a table is wrong, late, or drifting.
Level up the teams data work- Pair with and advise software engineers and data scientists on Spark, SQL, and modeling, and help turn notebook-grade code into production-grade pipelines.
Partner cross-functionally- Translate product, client, compliance, and business requirements into clear technical designs and dependable production systems.

Were looking for a Data Engineer with a strong applied ML focus to join our R&D department!
Why is this role so important?
Our Retail Intelligence products help leading brands and retailers understand how their products, brands and categories perform online. Behind them is data collected from retailers and marketplaces around the world: product pages, brands and categories, each described differently by every site.
Your mission is to turn that data into a single, trusted view: classifying products into a unified taxonomy, normalizing brands and attributes, and matching the same entities across sources. And doing it at scale, across a catalog of more than a billion records that keeps growing and changing every day.
This is an applied ML role within data engineering. Youll build with LLMs, agentic frameworks such as LangGraph, embeddings and classical ML, and ship them as production pipelines. Its hands-on work, not research for its own sake, but it takes a real understanding of classification and NLP methods to choose the right tool for each problem and prove that it works.
So, what will you be doing all day?
Your daily responsibilities may include:
Designing and building LLM-powered and ML-based pipelines that classify, normalize, structure and match product, brand and category data
Building agentic workflows (LangGraph or similar) that automate complex data tasks end to end
Choosing the right approach for each problem (LLMs, embeddings, fine-tuned models, classical classifiers or rules), balancing accuracy, cost and latency
Scaling solutions to run efficiently over billions of records, using Spark, Databricks and our cloud infrastructure
Building evaluation frameworks: ground-truth datasets, labeling processes, quality metrics and ongoing monitoring
Taking solutions from POC to production, and owning them after launch
Working closely with Product to define requirements and shape the roadmap
Collaborating with data engineers, data scientists and other R&D teams on infrastructure and best practices.

We're looking for an experienced Software Engineer to join our Data Platform Group. In this key role, you will build the company data platform: cloud-based microservices and data pipelines that process on the order of 1M records/sec at low latency. Because the platform is the foundation other groups build on, your work has a direct impact on our customers and enables engineering, product, and research teams across the organization.
The Lakehouse team owns the data itself - how it is stored, organized, retained, and served. We own our analytical storage layer, the batch processing built on top of it, and the APIs through which customers and the rest of the company consume data. If you enjoy the problems that only appear at petabyte scale - physical data layout, query performance, storage cost, and retention - this is the role.

Responsibilities:
End-to-end ownership of our large-scale analytical storage layer: data modeling, schema and table design, partitioning, retention, and query performance.
Design and develop the batch processing layer over our data lake using Spark on EMR (Java and PySpark).
Build and evolve Java/Spring Boot services that expose our data through well-defined APIs to customers and to consumers across the company.
Own performance and cost: query optimization, file layout and compaction, cluster sizing, and storage efficiency at scale.
Research new technologies in the lakehouse and analytical-storage space and adapt them for use in our product.
Work closely with product, DevOps, and security teams.