Netflix Case Study - Amazon Web Services (AWS) (2024)

Netflix, Inc. is an American technology and media services provider and production company headquartered in Los Gatos, California. Netflix was founded in 1997 by Reed Hastings and Marc Randolph in Scotts Valley, California. Netflixis a streaming service that offers a wide variety of award-winning TV shows, movies, anime, documentaries, and more on thousands of internet-connected devices.

Before starting the case study of Netflix, let's understand the meaning of AWS and cloud computing.

🔰What is AWS?

AmazonWeb Services (AWS) is the world's most comprehensive and broadly adoptedcloudplatform, offering over 175 services such as compute, databases, and storage. The platform is developed with a combination of infrastructure as a service (IaaS), platform as a service (PaaS) and packaged software as a service (SaaS) offerings.

🔰What is cloud computing?

Cloud computing is a term referred to storing and accessing data over the internet. It doesn't store any data on the hard disk of your personal computer. In cloud computing, you can access data from a remote server.From software and analytics tosecure and safe data storageand networking resources, everything can be delivered via the cloud.

For businesses, cloud computing means improved collaboration and productivity, as well as significant cost reductions. It means better data protection, improved availability, and expanded access to cutting-edge technologies.

🔰Types of cloud deployment

Public Cloud: A public cloud environment is owned by an outsourced cloud provider and is accessible to many businesses through the internet on a pay-per-use model. This deployment model provides services and infrastructure to businesses who want to save money on IT operational costs, but it’s the cloud provider who is responsible for the creation and maintenance of the resources.
Private Cloud: This cloud deployment model is a bespoke infrastructure owned by a single business. It offers a more controlled environment in which access to IT resources is more centralised within the business. This model can be externally hosted or can be managed in-house. Although private cloud hosting can be expensive, for larger businesses it can offer a higher level of security and more autonomy to customise the storage, networking and compute components to suit their IT requirements.
Hybrid Cloud: For businesses seeking the benefits of both private and public cloud deployment models, a hybrid cloud environment is a good option. By combining the two models, a hybrid cloud model provides a more tailored IT solution that meets specific business requirements.

🔰Cloud Services

There are three main service models of cloud computing – Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). There are clear differences between the three and what they can offer a business in terms of storage and resource pooling, but they can also interact with each other to form one comprehensive model of cloud computing.

IaaS (Infrastructure as Service)

This is the most common service model of cloud computing as it offers the fundamental infrastructure of virtual servers, network, operating systems and data storage drives. It allows for the flexibility, reliability and scalability that many businesses seek with the cloud, and removes the need for hardware in the office. This makes it ideal for small and medium sized organisations looking for a cost-effective IT solution to support business growth. IaaS is a fully outsourced pay-for-use service and is available as a public, private or hybrid infrastructure.

PaaS (Platform-as-a-Service)

This is where cloud computing providers deploy the infrastructure and software framework, but businesses can develop and run their own applications. Web applications can be created quickly and easily via PaaS, and the service is flexible and robust enough to support them. PaaS solutions are scalable and ideal for business environments where multiple developers are working on a single project. It is also handy for situations where an existing data source (such as CRM tool) needs to be leveraged.

SaaS (Software as a Service)

This cloud computing solution involves the deployment of software over the internet to variousbusinesses who pay via subscription or a pay-per-use model. It is a valuable tool for CRM and for applications that need a lot of web or mobile access – such as mobile sales management software. SaaS is managed from a central location so businesses don’t have to worry about maintaining it themselves, and is ideal for short-term projects.

✨ Netflix on AWS ✨

Netflix is the world’s leading internet television network, with more than 100 million members in more than 190 countries enjoying 125 million hours of TV shows and movies each day. Netflix uses AWS for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more—hundreds of functions that in total use more than 100,000 server instances on AWS.

🔰Why companies like Netflix select AWS❓

Today’s clear market leader for growing enterprises is Amazon Web Services (AWS), which began the cloud services trend in 2006 and has led the marketplace ever since. By integrating their systems with AWS, companies can focus on deploying and operating their own software instead of infrastructure and consume shared storage, computing and other resources faster and more efficiently.

Netflix,selected AWS because it provided Netflix with the greatest scale and the broadest set of services and features. They were also looking for a secure cloud path for their billing infrastructure and customer and employee data management systems. Today, AWS enables Netflix to quickly deploy thousands of servers and terabytes of storage within minutes. Users can stream Netflix content on PCs and most mobile devices from anywhere in the world.

🔹Netflix & Amazon Kinesis Data Streams Case Study🔹

🔴 🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴

Amazon Kinesis Data Streams processes multiple terabytes of log data each day, yet events show up in our analytics in seconds. We can discover and respond to issues in real-time, ensuring high availability and great customer experience.”

John Bennett

Senior Software Engineer, Netflix

🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴

🔰 What is Amazon Kinesis Data Streams (KDS)?

Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds to enable real-time analytics use cases such as real-time dashboards, real-time anomaly detection, dynamic pricing, and more.

Amazon Kinesis Data Streams enable real-time processing of streaming data at massive scale
Kinesis Streams enables building of custom applications that process or analyze streaming data for specialized needs

🔰 Kinesis Streams features

Handles provisioning, deployment, ongoing-maintenance of hardware, software, or other services for thedata streams
Manages the infrastructure, storage, networking, and configuration needed to stream thedata at the level of requireddata throughput
Synchronously replicates data across three facilities in an AWS Region, providing high availability and data durability
Storesrecords of a stream for up to 24 hours, by default, from the time they are added to the stream. The limit can be raised to up to 7 days by enabling extended data retention

🔰 Data Stream

Datastream represents a group of data records.
Data records in a data stream are distributed into shards.

🔹 Shard

Each shard provides has a sequence of data records
Streams are made of shards and are the base throughput unit of a Kinesis stream, aspricing is per shard basis.
Each shard supports up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys)
Each shard provides a fixed unit of capacity. If the limits are exceeded, either by data throughput or the number of PUT records, the put data call will be rejected with aProvisionedThroughputExceededexception.
This can be handled by:

Implementing a retry on the data producer side, if this is due to a temporary rise of the stream’s input data rate.
Dynamically scaling the number of shared (resharding) to provide enough capacity for the put data calls to consistently succeed

🔰 Kinesis Security

SupportsServer-side encryptionusing Key Management Service (KMS) for encrypting the data at rest
Supports writing encrypted data to a data stream byencrypting and decrypting on the client-side
Supportsinterface VPC endpointto keep traffic between VPC and Kinesis Data Streams from leaving the Amazon network. Interface VPC endpoints don’t require an IGW, NAT device, VPN connection, or Direct Connect.
Integrated with IAM to control access to Kinesis Data Streams resources.
Integrated with CloudTrail, which provides a record of actions taken by a user, role, or an AWS service in Kinesis Data Streams.

✨ From the outset, AWS enabled Netflix to experiment with different approaches to analyzing its network data. “Early in the design process, the flexibility to try different ways of processing the data was important,” says Bennett. “We experimented with multiple designs and used many AWS products to get here.”

The solution Netflix ultimately deployed—known internally as Dredge—centralizes flow logs usingAmazon Kinesis Data Streams. The application reads the data from Amazon Kinesis Data Streams in real-time and enriches IP addresses with application metadata to provide a full picture of the networking environment. “Usually, we would put the data into a database, which would build an index to enable faster querying,” says Bennett. “Dredge joins the flow logs with application metadata as it streams and indexes it without using a database, which eliminates a lot of the complexity.”

The enriched data lands in an open-source analytics application called Druid. Netflix uses the OLAP querying functionality of Druid to quickly slice data into regions, availability zones, and time windows to visualize it and gain insight into how the network is behaving and performing.

AWS was the logical choice for Dredge in part because the data was already resident in the AWS Cloud. “It would have been daunting to publish, stream, and consume that much information from an external system such as Kafka,” says Bennett. “It took just a few API calls to centralize multiple terabytes of flow logs into Amazon Kinesis Data Streams. Now we can focus on getting insights from the data rather than simply getting access to it.”

The scalability of Amazon Kinesis Data Streams was a good fit for the Dredge application because of the cyclical and elastic nature of network usage at Netflix. “When it comes to our networking data, it’s more cost-efficient to be able to scale up and down, which is not as easy to do with alternatives to Amazon Kinesis Data Streams,” says Bennett.✨

✨ Netflix’s Amazon Kinesis Data Streams-based solution has proven to be highly scalable, each day processing billions of traffic flows. Typically, about 1,000 Amazon Kinesis shards work in parallel to process the data stream.✨