Software Engineer (Jet.com, Hoboken, NJ)
Play a pivotal design & hands on implementation role in improving the Data infrastructure in a project-oriented work environment. Gather & process raw data at scale - collect data across all business domains (our functional-first, event sourced, micro services backend, Kafka) & expose mechanisms for large scale parallel processing. Design & implement a near real-time ingestion pipeline into a data warehouse & Hadoop data lake using spark streaming. Process unstructured data into a form suitable for analysis & then empower state-of-the-art analysis for analysts, scientists. Build efficient new Data Models & refactor existing ones. Partner with business to build right data models & analytics capabilities. Solve complex SQL & Big Data Performance challenges. Implement controls, policies, processes & best practices in the Data Engineering space. Help in improving database deployment & change management process. Provide reliable & efficient Data services as part of the database team. Work closely with the developers on development best practices & standards.
Minimum Requirements: Bachelors degree or equivalent in Computer Science, Information Technology, Engineering or related field & 5 years of experience processing large amounts of data. Experience with building data stream-processing systems using solutions such as Spark-Streaming. Experience with big data querying tools, such as Hive, Presto. Experience with messaging systems such as Kafka. Demonstrated knowledge of data structures, database management systems & distributed computing frameworks to be able to manage vast amounts of data in open source technology implementations. Demonstrated knowledge of software engineering & development techniques. Demonstrated knowledge of discrete structures & microservices architecture. Demonstrated knowledge of Lambda Architecture. Experience engineering data solutions using technologies including Spark, Scala/Java, HBase, Spark, Oozie, Yarn. Experience in building data pipelines, deploying & managing Big Data infrastructure, & establishing deployment & operational excellence of Big Data clusters. Experience with tuning & troubleshooting in large data environments on the Hadoop or RDBMS stack. Employer will accept any amount of experience with the required skills.
To Apply, click Apply. EOE. MFDV.