Required Machine Learning Scientist II - GenAI Evaluation
20717
Role Description:
As a Machine Learning Scientist, your work will focus on the evaluation and optimization of generative AI systems. You will develop and fine-tune Judge LLMs to assess model outputs across a variety of tasks, design robust evaluation frameworks for agentic workflows, and build scalable pipelines for synthetic data generation. The team also plays a critical role in multilingual evaluation, enabling GenAI applications to support market expansion across all supported languages.
Key Job Responsibilities and Duties:
Develop and apply state-of-the-art techniques for evaluating generative AI systems, with a focus on agent workflows, multilingual output, and task-specific Judge LLMs.
Design and implement scalable evaluation pipelines, including synthetic data generation and benchmarking for model quality, relevance, and consistency..
Optimize and maintain Judge LLMs to assess outputs across dialog systems, Q&A, and trip planning use cases.
Conduct in-depth data analysis to define and track evaluation metrics, validate label quality, and explore performance across different languages and user scenarios.
Ensure the reliability, efficiency, and scalability of evaluation tools and frameworks in both offline and online environments.
Collaborate closely with ML engineers to integrate evaluation components into production pipelines, supporting continuous improvement of GenAI applications.
Work cross-functionally with product, research, and analytics teams to align evaluation strategies with business goals and user impact.
Requirements: Advanced knowledge and experience in Computer Vision and Natural Language Processing, engineering aspects of developing ML and GenerativeAI models at scale.
Experience designing and executing end-to-end research and development plans and generating impact through large-scale machine learning model development. Preferably evidenced by peer-reviewed publication, patents, open sourced code or the like.
Relevant work or academic experience (MSc + 4 years of working experience, or PhD + 2 years of working experience), involved in the application of Machine Learning to business problems.
Masters degree, PhD or equivalent experience in a quantitative field (e.g. Computer Science, Engineering Mathematics, Artificial Intelligence, Physics, etc.).
Experience on multiple machine learning facets: working with large data sets, model development, statistics, experimentation, data visualization, optimization, software development.
Experience collaborating cross functionally in the development of machine learning products (e.g. Developers, UX specialists, Product Managers, etc.).
Strong working knowledge of Python, Java, Kafka, Hadoop, SQL, and Spark or similar technologies. Working experience with version control systems.
Excellent English communication skills, both written and verbal.
Successfully driving technical, business and people related initiatives that improve productivity, performance and quality while communicating with stakeholders at all levels
Leading by example, gaining respect through actions, not your title. Developing your team and motivating them to achieve their goals. Providing feedback timely and managing your key team performance indicators.
This position is open to all candidates.