The worlds largest content recommendation platform, is hiring a Senior Big Data Engineer for the Infrastructure Engineering R&D group.
What are some of the things you do on a day-to-day basis?
Develop one of the largest real time high scale operation in the world to support over 1.4B users, >500,000 requests / second, 140TB of new data every day
Have end to end ownership: Design, develop, deploy, measure and maintain our services
Solve production issues in the framework code
Collaborate with the brightest software engineers team members
Influence directly on the way billions of people discover the internet
Here are some of the things software devs in our group did in the last several months:
Rewrote part of our recommendation servers to reduce serving time p999 by >40%
Rolled out custom Cassandra compaction strategy
Created and improved our Kafka consumer designed for very high scale
Developed a brand new metrics pipeline, ingesting over 100B metric messages per day
Anomaly detection engine for application metrics
Rolled out a new custom home-built our authentication service
Many more things
Our Tech Stack:
Java, Spark, Kafka, Hadoop, Cassandra, Vertica, MySQL, HDFS, BigQuery, Docker, Linux, Kubernetes, Prometheus, Metrictank, Grafana, Airflow, Redis
This is us!
You will work with:
The people who wrote the first lines of code (and many many more lines after that)
The proud owner of a beer brewery
People who are deeply familiar with the code of Kafka, Cassandra, AirFlow, Prometheus and Spark (including committing back)
Famous internal hackathon winners
The designers and implementers of a offheap, zero GC, Custom Kafka Producer which cumulatively sends ~100B messages / day.
Requirements: Experience developing large scale distributed systems
Deep understanding of strong Computer Science fundamentals: object-oriented design, data structures systems, applications programming and multi threading programming
5+ years programming experience in Java or equivalent Object-Oriented language
Production systems understanding
Experience with SQL and No-SQL
Highly Proficient in Linux environment
Proven ability to work effectively across multiple teams
BS.c in computer science or equivalent experience
It would be great if you also have:
Experience in Big Data tools
Knowledge in algorithms, data-mining and machine learning.
This position is open to all candidates.