We're very excited to invited Dr. Jiang from Netflix to talk about how Netflix uses cutting edge features in Spark SQL such as predicate pushdown, & catalyst optimizations to compute the features for offline training pipeline while maintaining the consistency between online feature & offline feature generation.
Food & drink will be sponsored by Netflix. Join us.
Abstract:
Netflix is the worlds largest streaming service, with over 80 million members worldwide. Machine learning algorithms are used to recommend relevant titles to users based on their tastes.
At Netflix, we use Apache Spark to power our recommendation pipeline. Stages in the pipeline, such as label generation, data retrieval, feature generation, training, validation, are based on Spark ML PipleStage framework. While this provides developers the flexibility to develop individual components as encapsulated pipeline stages, we find that coordination across stages can potentially provide significant performance gains.
In this talk, we discuss how our machine learning pipeline based on Spark has been improved over the years. Techniques such as predicate pushdown, wide transformation minimization, have lead to significant run time improvement & resource savings.
Hua's Bio:
Hua Jiang received the Ph.D. degree in electrical engineering from the University of Minnesota, Twin Cities, in 2012. He was with the Design Group of Synopsys & the Data Infrastructure Group, LinkedIn Corporation. He is a Senior Software Engineer at the Personalization Group of Netflix His work includes building machine learning infrastructure & exploring for novel computational paradigms to accommodate fast-growing machine learning needs.
Agenda:
6:30 - 7:00 pm check-in/networking/food
7:00 pm -- 7:45 pm main talk
7:45 pm -- 8:00 pm Q/A
8:00 pm -- 8:30 pm closing
8:30 pm -- event/office closed
Here is the map of Netflix campus.