About the Role
If youre passionate about building large scale data processing systems, & you are motivated to make an impact in creating a robust & scalable data platform used by every team, come join us. You will jump into an early stage team that builds & manage a workflow orchestration service. You will help shape the vision & architecture of WeWork's next generation data infrastructure, making it easy for developers to build data-driven products & features. You are responsible for developing a reliable infrastructure that scales with the companys incredible growth. Your efforts will allow accessibility to business & user behavior insights, using huge amounts of WeWork data to fuel several teams such as Analytics, Data Science, Sales, Revenue, Product, Growth & many others as well as empowering them to depend on each other reliably. You will be a part of an experienced engineering team & work with passionate leaders on challenging distributed systems problems.
About the Mission
Data is at the core of our business, providing insights into the effectiveness of our products & enabling the technology that powers them. The mission of Developer Platform is to empower WeWork to build better products faster. Were building an ecosystem where consumers & producers of data can depend on each other safely. We thrive to build high-quality systems we can be proud to open source & an amazing experience for our users & ourselves. We regard culture & trust highly & are looking forward to welcoming your contribution to the team.
- Build & maintain high-performance, fault-tolerant, secure, & scalable fully managed workflow orchestration service built on top of Apache Airflow that empowers data platform across the entire company.
- Lead development of high leverage projects & capabilities of the platform
- Partner with engineers & product leaders to design & build robust features on Airflow to support all data engineering teams across WeWork.
- Thinking through long-term impacts of key design decisions & handling failure scenarios
- Form a holistic understanding of tools, key business concepts (data tables), & the data dependencies & team dependencies
- Building self-service platforms to power WeWorks Technology
- 7-9+ years of experience
- When thinking about Big Data workflow engines you are not just experienced with Apache Airflow. You know under the hood implementations on this platform.
- Strong Python & Unix development experience
- Solid experience with Docker, Kubernetes or other container technologies
- Write server-side code to power the Airflow platform with Python
- Build, configure & implement features against other open-source infrastructure components that empower Airflow platform, such observability(Prometheus & Grafana) & Kubernetes
- Experience shipping several high quality of complex software's release.
- Strong troubleshooting & performance tuning skills.
- Experience in development for cloud-based systems, ideally AWS
- Strong foundation in algorithms & data structures & their real-world use cases.
- Excellent communication (verbal & written) & interpersonal skills & an ability to effectively communicate with both business & technical teams
- Strong understanding of distributed systems concepts & principles (consistency & availability, liveness & safety, durability, reliability, fault-tolerance, consensus algorithms)
- Strong Experience with one or more of the following technologies:
- Workflow management - Luigi, Apache Oozie or Azkaban
- Distributed logging systems (Kafka, Pulsar, Kinesis, etc)
- MPP databases (Redshift, Vertica, )
- Cloud storage: S3, GCS
Nice to Have
- 3+ years experience in leadership roles or engineering management.
- Experience in open source development
- Data warehouse tooling
WeWork Technology is bridging the gap between physical & digital platforms, providing a delightful, flawless & powerful experience for members & employees. We build software & hardware that enables our members to connect with each other & the space around them like never before.
We augment our community & culture teams through the tools we build. We believe theres a macro shift toward a new way of workingone focused on a movement towards meaning & purpose.
WeWork Technology is proud to be shaping this movement.We are a team of passionate, fearless & collaborative problem-solvers distributed globally with one goal in mind - to humanize technology across the world.
We are an equal opportunity employer & value diversity in our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.