In your role as a Data Engineer, you'll be actively engaged in the most fascinating and demanding projects for our clients. Handling high-scale operations, heavy traffic, stringent SLAs, and massive data volumes is just the starting point. Our leadership in the realms of Big Data, Streaming Data, and Complex Event Processing (CEP) stems from our adept use of cutting-edge, optimal technologies.
Responsibilities:
Apply software-engineering methods and best practices (Medallion Architecture) in Data Lake management, using tools like DBT.
Develop Data pipelines to ingest and transform data.
Manage and enhance orchestrated pipelines across company
Data Modeling: Developing data models to ensure efficient storage, retrieval, and processing of data, often involving schema design for NoSQL databases, data lakes, and data warehouses. This will also include catalogs of data.
Scalability: Ensuring that the big data architecture can scale horizontally to handle large volumes of data and high traffic loads.
Data Security: Implementing security measures to protect sensitive data, including encryption, access controls, and data masking.
Data Governance: Establishing data governance policies to ensure data quality, compliance with regulations, and data lineage.
Requirements: 5 years of experience in Python (+Java / Scala)
Experience in Building data lakes, including ingestion and transformation (DBT)
Data Warehousing & ETL Pipelines
SQL & Advanced Query Optimization
Experience in data formats (Parquet, Avro, ORC)
Experience in streaming (Spark Streaming, Flink, Kafka Streams, Beam, etc.)
Experience in NoSQL and data storage (e.g. Elastic, Redis, MongoDB, CouchBase, BigQuery, Snowflake, Databricks)
Experience in messaging (Kafka, RabbitMQ, etc.)
Experience in cloud platforms (AWS, GCP)
This position is open to all candidates.