Senior Big Data Engineer
A+E Networks
New York, NYThis was removed by the employer on 6/2/2019 5:38:00 AM PST
Not to worry we have many other jobs on the site;
Browse all jobs
Browse the IS/IT Category
Search for Senior Big Data Engineer jobs in New York-NY
Search all Senior Big Data Engineer postings
Full Time Job
THE ROLE: Senior Big Data Engineer
A highly skilled individual who has a proven track record and takes pride in bringing high-quality product quickly to market, enjoys learning and applying new technologies, and thrives in a dynamic engineering-oriented work environment. Responsible for developing ETL pipeline for A E's Big Data initiatives. Managing backend development lifecycle including design, development, testing, and deployment. Research new technologies, methodologies, architecture patterns so as to select the correct solution depending on the need. Assess areas of re-use and architect solutions that can be easily integrated across products. Ensure that solutions are architected and implemented to scale. Write logic in the appropriate programming language to meet the functional requirements. Ensure the appropriate test cases are implemented. Ensure proper documentation of code before the code is deployed to production.
MORE ABOUT WHAT YOU'LL DO:
• Software development and design in Python
• Hands on experience with Apache Spark / Hadoop
• Deep expertise in SQL and ETL batch processing
• Proficient in a Unix/Linux, CentOS
• Hands on experience in AWS: S3 / EMR / Data Pipeline / Lambda
WHAT YOU'LL LEARN:
You will use your analytical skills, enthusiasm for telling stories through data and knowledge of economics and statistics to build A E Networks brand and credibility. You will work closely with the rest of the research team to respond to data requests, conduct analysis on digital trends, and build data that drives insights and results.
YOUR STORY:
Ideal candidates are self-starters who love a challenge and can work in a fast-paced, deadline-driven environment. The Engineering team works directly with the Managing Director for Data Mining & Analytics and the rest of the Data Science team to develop solutions.
• 1-2 years of relevant experience
• Bachelor's degree in computer science, data science, computer engineering or related field from an accredited university, master's degree a plus
• Proven experience with building and optimizing data pipelines, architectures, and datasets from both structured and unstructured datasets
• Demonstrated ability to build processes that support data transformation, structures, metadata, accuracy checking, and workload management
• Experience and proficiency in the following tools and technologies:
- Command line tools and git version control technology
- Parallel computing (e.g. Hadoop, Kafka, Spark, Hive, etc.) and multithreading
- Cloud-based services (Google Cloud, Amazon Web Services)
- Relational SQL and NoSQL databases, graphical database (e.g. Neo4J) experience a plus
- Programming languages (e.g., Python, R, Java, C , C#, Ruby, JavaScript, etc.)
- Front-end web development or visualization experience a plus (e.g. Tableau, D3js, leaflet, R Shiny, etc.)
- Systems Development Life Cycle (''SDLC'') best practices
- Interfacing with application programming interface (''API''), building interfaces a plus
- Specific ETL tools/schedulers (e.g. Alteryx, Dask, Airflow, Luigi, etc.) a plus
- Algorithms and data structures a plus
• Organized, self-directed, and resourceful with the ability to appropriately prioritize work in a fast-paced environment
• Excellent communication and visualization skills, with the ability to synthesize disparate data and information into a strong narrative
• Team player with an outgoing personality and high level of integrity
THAT SOMETHING EXTRA:
If you've read this far, you're likely a great fit for us…and maybe you're fluent in sign language, have studied the circus arts or are currently writing a screenplay. If you have a special skill or backstory that might directly or indirectly help you succeed in this role, we'd love to hear about it in your cover letter.