Four Reasons We Choose Amazon’s Cloud as Our Computing Platform

One year ago, none of Netflix’s customer traffic was supported out of AWS, Amazon’s cloud services. Today, most of our member traffic is supported by software we’ve built and deployed in AWS. This includes much of our member website at Netflix.com as well as the software supporting many Netflix Ready Devices, for example: the Xbox, PS3, Wii, AppleTV, iPhone, and iPad.

We’ve built and deployed search engines in AWS, recommendation systems, A|B testing infrastructure, streaming servers, encoding software, huge data stores, caching architectures, various SQL and NoSQL solutions, all in 2010 for the first time.

This has been a Herculean effort by our engineering teams, of re-architecture, quick learning, nimble coding, incredible effort and can-do attitude. The teams have managed this feat while at the same time supporting massive business growth throughout the year, adding thousands of new titles to our streaming library, and launching many new Netflix Ready Devices.

Our move to AWS will be the subject of various future posts to this blog, I wanted to kick the subject off with an explanation for why we chose AWS as a platform. This is a question that comes up fairly often when we interview engineering candidates.

1. We needed to re-architect, which allowed us to question everything, including whether to keep building out our own data center solution.

Netflix has been fortunate to experience incredible growth in recent years, both in terms of customers and streaming devices. We started to hit that point where every layer of our software stack needs to be able to scale horizontally. With the shift to streaming, our software needs to be much more reliable, redundant, and fault tolerant.

After many fruitful years of growth on our previous three tier, database oriented architecture, it was time for us to go back to the drawing board. We could have chosen to build out new data centers, build our own redundancy and failover, data synchronization systems, etc. Or, we could opt to write a check to someone else to do that instead.

2. Letting Amazon focus on data center infrastructure allows our engineers to focus on building and improving our business.

Amazon calls their web services “undifferentiated heavy lifting,” and that’s what it is. The problems they are trying to solve are incredibly difficult ones, but they aren’t specific to our business. Every successful internet company has to figure out great storage solutions, hardware failover, networking infrastructure, etc.

We want our engineers to focus as much of their time as possible on product innovation for the Netflix customer experience; that is what differentiates us from our competitors.

3. We’re not very good at predicting customer growth or device engagement.

Netflix has revised our public guidance for the number of subscribers we will end 2010 with three times over the course of the year. We are operating in a fast-changing and emerging market.

How many subscribers would you guess used our Wii application the week it launched? How many would you guess will use it next month? We have to ask ourselves these questions for each device we launch because our software systems need to scale to the size of the business, every time.

Cloud environments are ideal for horizontally scaling architectures. We don’t have to guess months ahead what our hardware, storage, and networking needs are going to be. We can programmatically access more of these resources from shared pools within AWS almost instantly.

4. We think cloud computing is the future.

One year ago, our cloud computing expertise was limited to research and some prior experience at other companies. Today Netflix is running one of the highest volume cloud computing deployments in the world. Engineers who choose to work at Netflix are developing skills that will be increasingly relevant over the years to come as cloud computing becomes the dominant platform for growing internet companies.

We believe this transition will take place. It will help foster a competitive environment for cloud service providers which will help keep innovation high and prices dropping. We chose to be pioneers in this transition so we could leverage our investment as we grow, rather than to double down on a model we expect will decline in the industry. We think this will help differentiate Netflix as a place to work, and it will help us scale our business.

Next up I’ll post on some of the mistakes we’ve made and lessons we’ve learned through our transition to AWS.

-john ciancutti.