Events  Classes  Deals  Spaces  Jobs 
    Sign in  
 
 
PulsePoint is ad tech fusing programmatic targeting, distribution & optimization w/ content marketing.
 
Engineering, Full Time    New York    Posted: Monday, November 19, 2018
 
   
 
Apply To Job
 
 
 
JOB DETAILS
 

PulsePoint Data Engineering team plays a key role in our technology company thats experiencing exponential growth. Our data pipeline processes over 50billion impressions a day (> 20TB of data, 220TB uncompressed). This data is used to generate reports, update budgets, & drive our optimization engines. We do all this while running against extremely tight SLAs & provide stats & reports as close to real-time as possible.

The most exciting part about working at PulsePoint is the enormous potential for personal & professional growth. We are always seeking new & better tools to help us meet challenges such as adopting proven open-source technologies to make our data infrastructure more nimble, scalable & robust. Some of the cutting edge technologies we have recently implemented are Kafka, Spark Streaming, Docker & Mesos.

What you'll be doing:

  • Design, build & maintain reliable & scalable enterprise level distributed transactional data processing systems for scaling the existing business & supporting new business initiatives
  • Optimize jobs to utilize Kafka, Hadoop, Vertica, Spark Streaming & Mesos resources in the most efficient way
  • Monitor & provide transparency into data quality across systems (accuracy, consistency, completeness, etc)
  • Increase accessibility & effectiveness of data (work with analysts, data scientists, & developers to build/deploy tools & datasets that fit their use cases)
  • Collaborate within a small team with diverse technology backgrounds
  • Provide mentorship & guidance to junior team members

TeamResponsibilities:

  • Installation, upkeep, maintenance & monitoring of Kafka, Hadoop, Vertica, RDBMS
  • Ingest, validate & process internal & third party data
  • Create, maintain & monitor data flows in Hive, SQL & Vertica for consistency, accuracy & lag time
  • Maintain & enhance framework for jobs(primarily aggregate jobs in Hive)
  • Create different consumers for data in Kafka such as flafka for Hadoop, flume for Vertica & Spark Streaming for near time aggregation
  • Train Developers/Analysts on tools to pull data
  • Tool evaluation/selection/implementation
  • Backups/Retention/High Availability/Capacity Planning
  • Disaster Recovery- We have all our core data services in another Data Center for complete business continuity
  • Review/Approval - DDL for database, Hive Framework jobs & Spark Streaming to make sure they meet our standards
  • 24*7 On call rotation for Production support

Technologies We Use:

  • Chronos - for job scheduling
  • Docker - Packaged container image with all dependencies
  • Graphite/Beacon - for monitoring data flows
  • Hive - SQL data warehouse layer for data in HDFS
  • Impala- faster SQL layer on top of Hive
  • Kafka- distributed commit log storage
  • Marathon cluster wide init for Docker Containers
  • Mesos - Distributed cluster resource manager
  • Spark Streaming - Near time aggregation
  • SQL Server - Reliable OLTP RDBMS
  • Sqoop - Import/Export data to RDBMS
  • Vertica - fast parallel data warehouse

Required Skills:

  • BA/BS degree in Computer science or related field
  • 5+ years of software engineering experience
  • Knowledge & exposure to distributed production systems i.e Hadoop is a huge plus
  • Proficiency in Linux
  • Fluency in Python, Experience in Scala/Java is a huge plus
  • Strong understanding of RDBMS, SQL;
  • Passion for engineering & computer science around data
  • Willingness to participate in 24x7 on-call rotation

What youll get:

  • Sane work hours (with flexible scheduling)
  • Competitive Salary & 401K Plan Match
  • Generous paid vacation (we consider your birthday a holiday)
  • Sabbatical at 5 years of employment
  • Health & Wellness Fairs
  • The opportunity to partake in our Office Fitness Shape-Up Program
  • Professional training & industry membership access
  • Annual Company Retreat
  • Complimentary membership to local programs like NYC CitiBike
  • Corporate Discount to New York Sports Club (NYSC)
  • Free team lunches twice a month
  • Team happy hours & beer-o-clock Fridays
  • Awesome snacks: drink bar, coffee bar, ice cream bar, candy bar & fruit bar
  • The opportunity to join our Company Basketball Team
  • Indoor dart wars, Ping-Pong Tournaments, walking desks, annual office olympics

Want to peek inside the Pulsepointoffices? Check it out here:https://www.themuse.com/profiles/pulsepoint

 
 
 
Apply To Job
 
 
 
 
 
© 2018 GarysGuide      About   Terms   Press   Feedback