Uber Engineering is excited to invite you to this open source meetup in Palo Alto, where we will highlight how we use Hadoop & Spark to manage data at a global scale. Presentations will cover how our Data Platform team handles the massive amount of data coming in from cities all over the world; development to improve Hadoops scalability in partnership with the open source community; solutions for operating Hadoop infrastructure across a large organization; & how we manage 80,000-plus Spark apps.
Join us to learn more about how Uber embraces open source technology & culture to create an open & innovative engineering organization.
Agenda
6:00pm -6:30pm - Doors open, food, & drinks
6:30pm -6:40pm - Unique data challenges at Uber scale- Suresh Srinivas
6:40pm -7:00pm - Future of HDFS at Uber- CR Hota & Chao Sun
7:00pm- 7:20pm- Managing Apache Hadoop at Scale-Mithun Mathew and
Liang Gao
7:20pm -7:35pm- Keeping Up With Apache Sparks Popularity at Uber-
Abhishek Modi
7:35pm -8:00pm- Q&A & Networking
Unique data challenges at Uber scale
Speaker: Suresh Srinivas
Uber's mission is to bring transportation for everyone & everywhere. This mission has powered exponential growth of Uber at a global scale, starting with ridesharing & expanding into many offerings, such as Uber Eats, Uber Freight, Uber for Business, Uber Delivery, & Uber Health. Data is at the core of Ubers business of creating great experiences & efficiencies for our users & partners. Scale of data coupled with the speed of data, & the need to derive insights & decisions faster, brings about unique challenges related to agility & technology. This talk will cover how the Data Platform team at Uber handles these unique challenges & key learnings from that work.
Future of HDFS at Uber
Speaker: CR Hota, Chao Sun
In the last few years, Uber's prime Hadoop cluster has grown to hold ~100PB worth of data. This volume of data, along with constraints to scale a single Hadoop cluster, has thrown new challenges in terms of scaling the current Hadoop infrastructure while still maintaining a single unified view of data. This talk will focus on how Uber's Hadoop team is designing a solution by working closely with the open-source community to tackle this problem of scale & efficiency. The talk will touch upon how various new features in Hadoop, such as Router-based federation, Observer Namenode, Erasure coding, & Tiering Service, along with interrelated projects will help shape Hadoop's future at Uber.
Managing Apache Hadoop at Scale
Speaker: Mithun Mathew, Liang Gao
Over the last three years Hadoop at Uber has grown from a footprint of 50 to 15,000+ servers. In this presentation, we focus on solutions we built targeting problems that arise from managing open source Hadoop in a large scale production environment. During the presentation we will cover three main topics:
Resource management system that empowers teams to manage their own resources & align them with organizational budget.
Our automated cluster deployment & management solution that has helped our team deploy & operate Hadoop clusters across 15,000+ servers.
Our future work on anomaly detection & auto-remediation as we continue to scale our infrastructure.
Keeping Up With Apache Sparks Popularity at Uber
Speaker: Abhishek Modi
There are 80,000 Spark apps that run at Uber every day. Lets talk about how Ubers Spark Compute Infra Service handles this load. In this talk, well discuss how Ubers Spark ecosystem started out as a number of different spark builds & semi-managed clusters, & how this evolved to today, where we have managed builds & clusters as well as automated monitoring, profiling, autotuning, & dependency management! Well also talk about the issues that we face today & how were planning on tackling these.
Please join our Uber Open Source facebook page for updates: https://www.facebook.com/uberopensource/
