Join us for an Apache Kafka meetup on December 5th at 5:30pm, hosted at Confluent's brand new office & event space in Mountain View! All details below|
1) RSVP below
2) Fill in this short form: https://docs.google.com/forms/d/e/1FAIpQLSfS2LCA1T6rbnE9QXVzcEHoVm-Kvo6Z6sAJZkSL_VTw84vSCA/viewform
3) Prior to the event, you will receive an email asking you to register for the event & sign an NDA, if you do that we'll have a badge ready for you when you arrive!
You can still attend if you do not fill the form above, though it may take slightly longer to check in.
5:30pm: Networking, Pizza & drinks!
6:00pm: Anna Povzner, Confluent
6:40pm: Manpreet Singh, Yelp
7:20-8pm: Additional Q&A & Networking
First Talk: Anna Povzner
Protecting Tenant Performance in Multi-tenant Kafka
Deploying Kafka to support multiple teams or even an entire company has many benefits. It reduces operational costs, simplifies onboarding of new applications as your adoption grows, & consolidates all your data in one place. However, this makes applications sharing the cluster vulnerable to any one or few of them taking all cluster resources. The combined cluster load also becomes less predictable, increasing the risk of overloading the cluster & data unavailability.
In this talk, we will describe how to use quota framework in Apache Kafka to ensure that a misconfigured client or unexpected increase in client load does not monopolize broker resources. You will get a deeper understanding of bandwidth & request quotas, how they get enforced, & gain intuition for setting the limits for your use-cases.
Anna Povzner is a software engineer on Cloud Native Kafka team at Confluent, & a contributor to Apache Kafka. Her main area of expertise is in resource management for performance SLAs & multi-tenancy in storage & distributed data systems. She received her Ph.D. from U.C. Santa Cruz, & was a researcher at IBM Almaden. Prior to Confluent, she was one of the early engineers in a storage startup where she helped build a scale-out content-addressable storage system from scratch.
Second Talk: Manpreet Singh
Running Large Scale Kafka Upgrades at Yelp
Abstract: Over the years at Yelp, we have relied on Kafka to build many complex applications & stream processing data-pipelines that solve a multitude of use cases, including powering our product experimentation workflow, search indexing, asynchronous task processing & more. This session will focus on the challenges we encountered & how we evolved our infrastructure tooling & upgrade strategy to overcome them. I will be talking about: How we rolled out new features such as kafka offset storage, message timestamp, reassignment auto-throttling, etc. Core technical issues discovered during upgrades such as failure of log cleaners due to large offsets while upgrading. The in-house test-suite that we built in order to: validate new kafka versions against our existing tooling & client-libraries, exercise the upgrade & rollback process & benchmark performance. The automation we built for safe & fast rolling upgrades & broker configuration deployment.
Manpreet Singh is a Software Engineer at Yelp who has led multiple infrastructure projects. He & his teammates are responsible for building & maintaining Yelps streaming & stream processing infrastructure using Kafka, handling billions of messages & terabytes of data per day. In his time at Yelp, he has designed, built & automated tooling for Kafka cluster rebalancing & has spearheaded the efforts to upgrade Kafka clusters & isolate revenue critical traffic to dedicated clusters. Manpreet is currently working on rearchitecting the deployment of Kafka Clusters to better cater to Yelps ever-growing scale.
NOTE: We are unable to cater for any attendees under the age of 21.