לוח משרות דרושים איש ביג דאטה Big Data משרה מלאה, לחיפוש זה נמצאו 31 משרות

We are looking for a Data Engineer to join our usage metering & billing platform team! with a strong background in writing ETLs and complicated data processing pipelines. Youre deeply familiar with modern data processing stack (especially Spark, Hadoop and Airflow), have a strong problem-solving attitude and, most importantly, youre passionate about understanding the business side of the platform. Our platform engineers have an end-to-end understanding of the data our products generate and the commercial models and data structures in which they fit.
Working with this enormous volume of data begins with massive ETL pipelines built using cutting-edge technologies such as Kafka, BigQuery, Airflow, S3, PostgreSQL, Athena, SQS, Redis, and especially Spark. We are big believers in working with the right tool for the right job, including programming languages. We run a truly polyglot shot with Scala, Python, Go and Node being especially common here . The extreme growth rate of our data and client base means that solutions need to be designed to handle 10X growth or more.
What youll do:
Develop end-to-end ETLs and data infrastructure features from data processing logic to database choice and implementation through to API development
Improve the scalability and performance of existing data pipelines
Delve into existing pipelines and business logic, target weaknesses and add functionality
Mentor, advise, and work collaboratively within the Biz Data & Dev group, with other development teams, and with all other business teams of the company
Continuously learn and follow new technologies both for personal growth and to ensure we are making the most informed technology choices possible

We are responsible for building data pipelines end to end-from raw data ingestion to the creation of actionable datasets-following the bronze, silver, and gold paradigm. This includes business logic, infrastructure, ETLs, optimization, and ongoing maintenance.

The data we deliver drives insights and decision-making across the organization and enhances our product offerings. We leverage technologies such as AWS, Snowflake, Iceberg, Airflow, Spark, and more.

What Youll Do
Lead the translation of business and product requirements into scalable data models, transformations, and pipelines.

Design and own datasets across bronze, silver, and gold layers, including defining grain, aggregations, and data contracts.

Develop and maintain SQL-heavy data pipelines and Airflow DAGs (workflow logic, dependencies, backfills, python, and lots of SQL).

Own data correctness for key business metrics (e.g., ARR), including deep root cause analysis and resolution of data issues.

Define and drive best practices for SQL, data modeling, and pipeline design across the team.

Optimize queries and data models for performance, scalability, and cost efficiency.

Collaborate closely with product managers, analysts, and BI developers to refine requirements and ensure high-quality data delivery.

Develop AI-agents to accelerate data analysis by internal and external users.

Work with complex data inputs (e.g., JSON, schemas, logs) and incorporate them into robust data pipelines.

Were looking for a Data Engineering Team Lead to take ownership of Ports data engineering team and help shape the future of agentic engineering portals. This is a hands-on leadership role: youll guide a team, drive technical direction, and help establish Ports data warehouse both for product use cases and for internal business analytics.

What youll do:

were building a platform by developers, for developers. As a Data Engineering Team Lead, youll balance leadership and data engineering work - setting technical direction while enabling your team to thrive.

Your responsibilities will include:

Lead the design and development of scalable and efficient data warehouse and BI solutions that align with organizational goals and requirements.
Utilize advanced data modeling techniques to create robust data structures supporting reporting and analytics needs.
Implement ETL/ELT processes to assist in the extraction, transformation, and loading of data from various sources into a shared data warehouse.
Identify and address performance bottlenecks within our data warehouse, optimize queries and processes, and enhance data retrieval efficiency.
Develop and provide a data backend for product facing features such as usage analytics, data insights and enabling an agentic context lake
Collaborate with cross-functional teams (product, R&D, analysts) to deliver actionable data solutions tailored to their needs.

we are looking for a Big Data Engineer.
As a Senior Big Data Engineer, working within Mobility Group, you will play a pivotal role in designing, developing, and maintaining the data infrastructure that powers our location analytics platform.
RESPONSIBILITIES:
Data Pipeline Architecture and Development: Design, build, and optimize robust and scalable data pipelines to process, transform, and integrate large volumes of data from various sources into our analytics platform.
Data Quality Assurance: Implement data validation, cleansing, and enrichment techniques to ensure high-quality and consistent data across the platform.
Performance Optimization: Identify performance bottlenecks and optimize data processing and storage mechanisms to enhance overall system performance and reduce latency.
Cloud Infrastructure: Work extensively with cloud-based technologies (GCP and AWS), to design and manage scalable data infrastructure.
Collaboration: Collaborate with cross-functional teams including Data Analysts, Data Scientists, Product Managers, and Software Engineers to understand requirements and deliver solutions that meet business needs.
Data Governance: Implement and enforce data governance practices, ensuring compliance with relevant regulations and best practices related to data privacy and security.
Monitoring and Maintenance: Monitor the health and performance of data pipelines, troubleshoot issues, and ensure high availability of data infrastructure.
Mentorship: Provide technical guidance and mentorship to junior data engineers, fostering a culture of learning and growth within the team.