Big Data Engineer
Powering their industry leading services requires highly scalable, available, reliable, secure, and performance systems. Their cloud and platform engineering teams build the infrastructure for their web crawling services. From Kubernetes to Machine Learning we they continually striving to push the envelope to bring the best value to their customers. Our client is looking for enthusiastic and passionate engineers to join their team. If you love working to bring real world value to many of the world’s largest companies, I would love to hear from you. Their platform runs thousands of jobs and processes millions of requests each day as they strive to collect the most accurate information about the way their customers products are used and consumed around the globe.
Our client looking for a Big Data Engineer to be a part of our Software Engineering team. In this role, you will be delivering software and striving for operational excellence for their sophisticated web crawling solutions and the processing pipeline. This is a unique opportunity to learn and work with other talented engineers working in a dynamic environment leveraging many of the latest technologies in the industry.
- Spend your time doing 20% architecture and design, 60% ETL and data cleansing, 20% big data analysis
- Enhancing, maintaining and curating our big data platform to enable our data analysts to productively work with extracted and curated data within the Hadoop and Google Bigquery environment
- Respond to requests to pull data from various sources into usable form.
Required Education and Experience:
- BS/MS in Computer Science or Mathematics, equivalent work experience
- You have 6+ years of experience in developing enterprise level software
- A solid understanding of Object Oriented Programming, Database Operations and Data Quality engineering
- Strong understanding of high traffic environments with at least 1 terabyte of data
- You are proficient in Java or Scala, Python, Node.js, regular expressions
- You are proficient in SQL, OLAP, Hadoop, HDFS, Hive, Spark, Hbase
- You have experience with ETL, Data Cleansing, ERD design, Data design, data performance optimization
- Experience developing in an Agile/SCRUM environment preferred
- Full Medical