Data Engineer

Who you are

• This role is a great fit for someone who is excited about working with large-scale, complex data, who cares about getting the details right, and who wants to grow under the mentorship of an experienced data engineering team
• Experience in data engineering
• Experience with data modeling, warehousing and building ETL pipelines
• Experience with AWS technologies like Redshift, S3, AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions
• Experience with non-relational databases / data stores (object storage, document or key-value stores, graph databases, column-family databases)

What the job involves

• When you order items on Amazon, there are practically thousands of ways we can fulfill that order
• Which Fulfillment Center do we ship from, what carriers do we use, what boxes do we combine items in — now scale that question to billions of items shipped annually worldwide
• At Amazon's Supply Chain Optimization Technologies (SCOT), we are tasked with fulfilling every customer order in the most intelligent way possible while making sure customers get their orders on time
• Within SCOT, the Fulfillment Optimization organization owns the artificial intelligence, optimization, and simulation systems that decide how Amazon's outbound network is planned and operated. Our team is a new horizontal team responsible for the data foundations that power every Fulfillment Optimization product — from long-range planning to day-of execution
• Our customers are the Applied Scientists, Business Intelligence Engineers, Data Scientists, and Software Development Engineers building those products, and the leaders who depend on trustworthy data to make multi-billion-dollar planning decisions
• We are looking for a Data Engineer to help us build that foundation. You will work alongside more senior Data Engineers, Software Development Engineers, and Applied Scientists to design, build, and operate the data pipelines, models, and warehouses that our products and analytics depend on
• You will take well-defined requirements, build a solution, and deliver it on schedule. You will learn the team's data architecture deeply, contribute to its evolution, and grow into an autonomous owner of significant pieces of it
• Build and optimize physical data models and ETL pipelines for datasets that serve our products and analytics, using Amazon's data platforms (Redshift, EMR, Spark, S3, Hive, etc.)
• Take well-defined requirements from Data Scientists, Business Intelligence Engineers, and Software Development Engineers, and deliver tested, documented, and maintainable data solutions on schedule
• Troubleshoot, root-cause, and resolve issues in existing datasets and pipelines, leaving them better and easier to maintain than you found them
• Measure and improve dataset quality — completeness, freshness, correctness — and contribute to the team's monitoring and SLA framework
• Write secure, stable, testable, and well-documented code in SQL and Python (or Scala), and submit it for code review
• Classify, store, and handle data in accordance with Amazon's security and privacy policies
• Participate in team design, scoping, and prioritization discussions; learn the business context behind the team's data architecture and the products it supports
• Collaborate with peers across Fulfillment Optimization and partner teams to integrate data sources and unblock downstream consumers
• You will spend your day writing SQL and Python, designing tables and pipelines, reviewing teammates' code, and pairing with data scientists and engineers to understand what they need from the data
• You will start with well-scoped pieces of work — a new dataset, an optimization to an existing pipeline, an investigation into a data quality issue — and grow your scope as you build context. You will be supported by a team that values mentorship, code review, and writing things down
• The Fulfillment Optimization organization owns and operates the artificial intelligence, optimization, and simulation systems that plan and run Amazon's outbound fulfillment network
• The organization is spread across the United States and Europe and is multi-disciplinary — Research Science, Applied Science, Business Intelligence, Product Management, Data Engineering, and Software Development
• Our team is a newly formed horizontal team responsible for the data foundations that all Fulfillment Optimization products and analytics depend on. Come join us as we build that foundation from the ground up