Senior Big Data Engineer
A+E NetworksNew York, NY
Full Time Job
THE ROLE: Senior Big Data Engineer
A highly skilled individual who has a proven track record and takes pride in bringing high-quality product quickly to market, enjoys learning and applying new technologies, and thrives in a dynamic engineering-oriented work environment. Responsible for developing ETL pipeline for A E's Big Data initiatives. Managing backend development lifecycle including design, development, testing, and deployment. Research new technologies, methodologies, architecture patterns so as to select the correct solution depending on the need. Assess areas of re-use and architect solutions that can be easily integrated across products. Ensure that solutions are architected and implemented to scale. Write logic in the appropriate programming language to meet the functional requirements. Ensure the appropriate test cases are implemented. Ensure proper documentation of code before the code is deployed to production.
MORE ABOUT WHAT YOU'LL DO:
• Software development and design in Python
• Hands on experience with Apache Spark / Hadoop
• Deep expertise in SQL and ETL batch processing
• Proficient in a Unix/Linux, CentOS
• Hands on experience in AWS: S3 / EMR / Data Pipeline / Lambda
WHAT YOU'LL LEARN:
You will use your analytical skills, enthusiasm for telling stories through data and knowledge of economics and statistics to build A E Networks brand and credibility. You will work closely with the rest of the research team to respond to data requests, conduct analysis on digital trends, and build data that drives insights and results.
Ideal candidates are self-starters who love a challenge and can work in a fast-paced, deadline-driven environment. The Engineering team works directly with the Managing Director for Data Mining & Analytics and the rest of the Data Science team to develop solutions.
• 1-2 years of relevant experience
• Bachelor's degree in computer science, data science, computer engineering or related field from an accredited university, master's degree a plus
• Proven experience with building and optimizing data pipelines, architectures, and datasets from both structured and unstructured datasets
• Demonstrated ability to build processes that support data transformation, structures, metadata, accuracy checking, and workload management
• Experience and proficiency in the following tools and technologies:
- Command line tools and git version control technology
- Parallel computing (e.g. Hadoop, Kafka, Spark, Hive, etc.) and multithreading
- Cloud-based services (Google Cloud, Amazon Web Services)
- Relational SQL and NoSQL databases, graphical database (e.g. Neo4J) experience a plus
- Front-end web development or visualization experience a plus (e.g. Tableau, D3js, leaflet, R Shiny, etc.)
- Systems Development Life Cycle (''SDLC'') best practices
- Interfacing with application programming interface (''API''), building interfaces a plus
- Specific ETL tools/schedulers (e.g. Alteryx, Dask, Airflow, Luigi, etc.) a plus
- Algorithms and data structures a plus
• Organized, self-directed, and resourceful with the ability to appropriately prioritize work in a fast-paced environment
• Excellent communication and visualization skills, with the ability to synthesize disparate data and information into a strong narrative
• Team player with an outgoing personality and high level of integrity
THAT SOMETHING EXTRA:
If you've read this far, you're likely a great fit for us…and maybe you're fluent in sign language, have studied the circus arts or are currently writing a screenplay. If you have a special skill or backstory that might directly or indirectly help you succeed in this role, we'd love to hear about it in your cover letter.
''THE DIVISION'S STORY
Were you the first kid on your block to get a PlayStation? Did you foresee the day when movies could be filmed on a phone and TV shows could be viewed on a wristwatch? The Digital team is made up of just such visionaries and early adaptors. We dream up new ideas for gaming, apps, and SVOD initiatives and generate engaging online content for our owned and operated websites. But if all that tech talk makes you glaze over, there's always 45th & Dean, our in-house digital content storytelling hub. It's a collective of talented writers, editors, motion graphics artists and producers who create content across all social media platforms as well as support our Ad Sales and Distribution teams. Our Editorial function, shared with Marketing, brings the voice, taste, and energy that brings our brands to life beyond linear.