Required Senior Data Engineer
Full-time
Company Description:
A dynamic technology firm dedicated to building exceptional products using modern technologies for the world's most admired companies. We pride ourselves on innovation, collaboration, and delivering excellence in every project.
Responsibilities:
Design and maintain scalable data pipelines with Spark Structured Streaming.
Implement Lakehouse architecture with Apache Iceberg.
Develop ETL processes for data transformation.
Ensure data integrity, quality, and governance.
Collaborate with stakeholders and IT teams for seamless solution integration.
Optimize data processing workflows and performance.
Additional Information:
Location: Ra'anana

We are looking for a Senior Data Engineer responsible for designing, building, and maintaining the infrastructure necessary for analyzing large data sets. This individual should be an expert in data management, ETL (extract, transform, load) processes, and data warehousing and should have experience working with various big data technologies, such as Hadoop, Spark, and NoSQL databases. In addition to technical skills, a Senior Data Engineer should have strong communication and collaboration abilities, as they will be working closely with other members of the data and analytics team, as well as other stakeholders, to identify and prioritize data engineering projects and to ensure that the data infrastructure is aligned with the overall business goals and objectives.

What You'll Do:
Work closely with data scientists/analytics and other stakeholders to identify and prioritize data engineering projects and to ensure that the data infrastructure is aligned with business goals and objectives.
Design, build and maintain optimal data pipeline architecture for extraction, transformation, and loading of data from a wide variety of data sources, including external APIs, data streams, and data stores.
Continuously monitor and optimize the performance and reliability of the data infrastructure, and identify and implement solutions to improve scalability, efficiency, and security.
Stay up-to-date with the latest trends and developments in the field of data engineering, and leverage this knowledge to identify opportunities for improvement and innovation within the organization.
Solve challenging problems in a fast-paced and evolving environment while maintaining uncompromising quality.
Implement data privacy and security requirements to ensure solutions comply with security standards and frameworks.
Enhance the team's dev-ops capabilities.

You will take ownership of the search infrastructure for Ask us - a financial intelligence platform powered by LLMs. Your main goal is to ensure our AI agents can find the exact financial data they need (articles, transcripts, news) in milliseconds. You will work on the Retrieval layer of our RAG architecture, combining traditional text search with modern vector search techniques.

Tech Stack: Elasticsearch (v8+), Python (FastAPI, Asyncio), OpenAI Embeddings, LangChain, LangSmith.

What You'll Do:
Search Engine Development: Design and implement Hybrid Search strategies. You will figure out how to best combine "keyword matching" (finding specific tickers like 'AAPL') with "semantic search" (finding concepts like 'revenue growth').
Relevance Tuning: You are responsible for the quality of search results. You will build systems to measure and improve how well the search engine answers user queries (using tools like LangSmith).
Vector Search & RAG: Manage the integration of OpenAI embeddings into Elasticsearch. You will solve challenges related to indexing long documents (e.g., earnings transcripts) so the AI retrieves only the most relevant parts.
Performance Optimization: Optimize Elasticsearch queries and index settings to ensure low latency, even for complex queries with many filters.
Python Backend: Develop and maintain the Python services that build queries and process results. We use FastAPI and Asyncio heavily.