As a Senior Big Data Engineer, working within our Mobility Group, you will play a pivotal role in designing, developing, and maintaining the data infrastructure that powers our location analytics platform.
RESPONSIBILITIES:
Data Pipeline Architecture and Development: Design, build, and optimize robust and scalable data pipelines to process, transform, and integrate large volumes of data from various sources into our analytics platform.
Data Quality Assurance: Implement data validation, cleansing, and enrichment techniques to ensure high-quality and consistent data across the platform.
Performance Optimization: Identify performance bottlenecks and optimize data processing and storage mechanisms to enhance overall system performance and reduce latency.
Cloud Infrastructure: Work extensively with cloud-based technologies (GCP and AWS), to design and manage scalable data infrastructure.
Collaboration: Collaborate with cross-functional teams including Data Analysts, Data Scientists, Product Managers, and Software Engineers to understand requirements and deliver solutions that meet business needs.
Data Governance: Implement and enforce data governance practices, ensuring compliance with relevant regulations and best practices related to data privacy and security.
Monitoring and Maintenance: Monitor the health and performance of data pipelines, troubleshoot issues, and ensure high availability of data infrastructure.
Mentorship: Provide technical guidance and mentorship to junior data engineers, fostering a culture of learning and growth within the team.
Requirements: REQUIREMENTS:
Strong hands-on Apache Spark experience - building and operating pipelines in production, not just familiarity.
Proficiency in PySpark or Scala for Spark development.
Proven track record delivering ETL pipelines and data integration at scale.
Solid SQL skills and command of data modeling concepts.
Cloud platform experience (AWS, GCP, or Azure) in a production data context.
Comfortable working with distributed systems and big data formats (Parquet, Delta Lake).
Nice to have:
Experience with pipeline orchestration tools, particularly Apache Airflow.
Exposure to the geospatial or location analytics domain.
Familiarity with Hadoop ecosystem components.
Background in both Python and Scala (beyond Spark context).
This position is open to all candidates.