Getting to know
Scalability
BITS Pilani Dr. Shreyas Rao
Associate Prof. (Off Campus), CSIS, BITS-Pilani
Pilani Campus
Instructor Profile
Dr. Shreyas Rao
• 18+ Years of Experience in IT, Teaching and Research
• Working as Associate Professor (Off Campus), Dept. of CSIS, BITS-Pilani, WILP
• B.E from VTU, M.S in Software Systems from BITS (WILP) and PhD from MAHE
• Worked as Business Analyst and Team Lead at SLK Software Services for 7+
years
• Previously worked in Presidency University and Sahyadri College, Mangaluru as
R&D Head, CSE
• Executed 10+ Consultancy projects
• COE member in AI&ML and COE member in Data Science (Govt. Sponsored for
1.2 Cr)
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
SE ZG583, Scalable Services
Lecture No. 1
Course Objectives
No Course Objective
CO1 Build competence to design, develop, implement and manage scalable information
systems
CO2 Gain understanding of different techniques & tools for building and managing
scalable services
CO3 Gain understanding of challenges and best practices in creating and managing
scalable services
BITS Pilani, Pilani Campus
Text Books and Reference
BITS Pilani, Pilani Campus
Evaluation
Evaluation Name Type Weight Duration Day, Date,
Component (Quiz, Lab, Project, (Open book, Session, Time
Midterm exam, End Closed book,
semester exam, etc) Online, etc.)
EC – 1 Quiz 1 5% February 13-23,
2023
Quiz 2 5% March 20-30, 2023
Lab (exploring different 10% To be announced
tools)
Assignment (end to 10% To be announced
end app development)
EC – 2 Mid-term Exam Open book 30% 2 hours Saturday,
11/03/2023 (FN)
EC – 3 End Semester Exam Open book 40% 2 ½ hours Saturday,
20/05/2023 (FN)
BITS Pilani, Pilani Campus
Flipkart Big Billion Dollar Day Sale
• Launched on 6-Oct-2014
• Opened at 8am
• Big discount in 70+ categories, flash sales, lucky draw
• Sold large units of Nokia Lumia 525, Samsung Galaxy Tabs
at throw-away prices
• Three Lakh order in 6 hours!
BITS Pilani, Pilani Campus
Flipkart Big Billion Dollar Day Sale
Negatives:
• Website crashed due to huge traffic and footfalls
• Already selected products vanished from cart after
recovery or appeared as sold out!
• Money got deducted from account, but order not executed
• Big complaints from customers
• Reviews were hidden; no refund and no cancellation of
orders
From an Architecture perspective, what could have gone wrong?
BITS Pilani, Pilani Campus
Hotstar Case Study
What steps did the Hotstar Cloud Architects take to handle 25.3 Million
concurrent users?
BITS Pilani, Pilani Campus
Current Day Scenario!
Daily Micro-Deployments
Company Deployments Streaming / OTT
Amazon 23,000/ day
Google 5,500 / day
Netflix 500 / day
Twitter 3 / week
* Source: Phoenix Project (2021)
Social Media Apps E-Commerce Apps
BITS Pilani, Pilani Campus
Software Quality Attributes
• Scalability
• Performance
• Availability
• Reliability
• Interoperability
• Testability
• Usability
• Modifiability
• Security
• Portability
• Maintainability
BITS Pilani, Pilani Campus
Agenda
1. What is Scalability
2. Need for Scalable Architectures
3. Principles of Scalability
4. Scale Cube
5. CAP Theorem
6. Guidelines for Building Highly Scalable Systems
7. Architecture’s Scalability Requirements
8. Challenges for Scalability
9. Case Study – Uber / Netflix
BITS Pilani, Pilani Campus
What is Scalability
• Scalable is the term used to describe software systems that can
accommodate growth
• It is a non-functional requirement
• Capability to handle growth in “some dimension” of its operations
Operational dimensions can include
• number of simultaneous users or requests a system can process
• amount of data a system can effectively process and manage
BITS Pilani, Pilani Campus
Scale Up vs Scale Down
• Scaling Up refers to increasing the size or capacity of a system
• Scaling Down refers to reducing the size or capacity of a system
Ex: Adding more Physical Servers, RAM, CPUs, Virtual Machines,
Containers etc.
BITS Pilani, Pilani Campus
Vertical vs Horizontal Scaling
• Vertical Scaling (Scaling Up) is defined as increasing a single machine’s capacity with
the rising resources in the same logical server or unit
• Horizontal Scaling (Scaling Out) is an approach to enhance the performance of the
server node by adding new instances of the server to distribute the workload equally
BITS Pilani, Pilani Campus
Vertical vs Horizontal Scaling
• Vertical Scaling - Adding more power to an existing ec2 instance
• Horizontal Scaling - Adding more ec2 instances
BITS Pilani, Pilani Campus
Need for Scalable Architectures
The foundations of scale need to be built in from the beginning, with the
recognition that the components will evolve over time.
“Hyper scalable systems exhibit exponential growth in computational
and storage capabilities while exhibiting linear growth rates in the costs
of resources required to build, operate, support and evolve the required
software and hardware resources.”
- Ion Gorton
BITS Pilani, Pilani Campus
Scalability and Costs
Ex: We have a Web-based (e.g. web server and database) system that can
service a load of 100 concurrent requests with a mean response time of 1
second. As the request load increases, we see the mean response time
steadily grow to 10 seconds with the projected load.
Requirement: Scale up this system to handle 1000 concurrent requests with the
same response time??
Solution:
Scale Up - running the Web server on a more powerful machine (Vertical)
Scale Out - run multiple instances of the Web server to increase capacity
(Horizontal, Cloning)
Is it that easy to scale?
BITS Pilani, Pilani Campus
Scalability and Costs (contd)
Some Potential causes that increase effort and cost
1) Database becomes less responsive with 1000 requests per second - Requires
upgrading the database server
2) Web server may generate slow dynamic requests (depends on other downstream
services) - requires identifying bottlenecks and scaling them
3) Web server framework that was selected emphasized ease of development over
scalability - complete rewrite is required
All the above requires development/upgradation cost that is seriously significant!!!
If a system is not designed intrinsically to scale, then the downstream costs and
resources of increasing its capacity to meet requirements may be massive
BITS Pilani, Pilani Campus
Tenets of Scalable Architecture
Architectures evolve over time.
1. Avoid single point of failure: Have multiple application instances (Ex: ec2)
2. Scale horizontally, not vertically
3. Stateless - no client context stored on the server
4. Caching - Well-managed caching eliminates some client–server interactions,
improving scalability and performance
5. Loose coupling - Targeted, granular changes are possible simplifying scaling
approaches
6. Role of observability - Scaling a system requires understanding its behavior.
Logs, metrics, and traces all give you the information
7. Use of APIs
All the above target – Availability / Performance and Reliability areas
BITS Pilani, Pilani Campus
Twelve Architectural Principles
1. N+1 Design. Never less than two of anything, and remember the rule of three.
2. Design for Rollback. Ensure you can roll back any release of functionality.
3. Design to Be Disabled. Be able to turn off anything you release.
4. Design to Be Monitored. Think about monitoring during design, not after.
5. Design for Multiple Live Sites. Don’t box yourself into one-site solutions.
6. Use Mature Technologies. Use things you know work well.
7. Asynchronous Design. Communicate synchronously only when absolutely necessary.
8. Stateless Systems. Use state only when the business return justifies it.
9. Scale Out Not Up. Never rely on bigger, faster systems.
10.Buy When Non Core. If you aren’t the best at building it and it doesn’t offer competitive
differentiation, buy it.
11. Commodity Hardware. Cheaper is better most of the time.
12. Design for at Least Two Axes. Think one step ahead of your scale needs.
Ref: R2 – The Art of Scalability
BITS Pilani, Pilani Campus
Twelve Architectural Principles
Ref: R2 – The Art of Scalability
BITS Pilani, Pilani Campus
Scale Cubes: 3 ways to scale an
application
The scale cube defines
three separate ways to
scale an application:
X-axis scaling load
balances requests across
multiple, identical
instances;
Z-axis scaling routes
requests based on an
attribute of the request;
Y-axis functionally
decomposes an application
into services.
Source: “Microservices Patterns” by Chris Richardson Book
BITS Pilani, Pilani Campus
Scale Cubes
• X-axis application splits scale linearly with transaction growth. They do not help with the growth in
code complexity, customers, or data. X-axis splits are “clones” of each other.
• Y-axis application splits help scale code complexity as well as transaction growth. They are mostly
meant for code scale because as they are not as efficient as x-axis in transaction growth.
• Y-axis splits tend to be more costly to implement than x-axis splits as a result of engineering time
necessary to separate monolithic code bases.
• Y-axis splits aid in fault isolation.
• Z-axis application splits help scale customer growth, some elements of data growth and
transaction growth.
• As with y-axis splits, z-axis splits aid in fault isolation.
• The choice of when to use what method or axis of scale is both art and science.
• Production data should be used over time to help inform the decision.
Source: “Microservices Patterns” by Chris Richardson Book
BITS Pilani, Pilani Campus
Scale Cubes: X-Axis Scaling
Source: “Microservices Patterns” by Chris Richardson Book
BITS Pilani, Pilani Campus
Scale Cubes : Z-Axis Scaling
Source: “Microservices Patterns” by Chris Richardson Book
BITS Pilani, Pilani Campus
Scale Cubes : Y-Axis Scaling
Source: “Microservices Patterns” by Chris Richardson Book
BITS Pilani, Pilani Campus
Scale Cubes: Three-Axis Split
BITS Pilani, Pilani Campus
CAP Theorem
CAP Theorem states that a distributed system can only guarantee two out of the three
characteristics: Consistency, Availability and Partition Tolerance
• Consistency is the ability for the system to
read from any replicated service instance,
and always return the most recent write
made across replicated services of the
same type.
• Availability is the ability for a service to
execute a request and receive a response
within a reasonable time interval without
timing-out or throwing an error.
• A network partition occurs when one or
more of the application's services become
unreachable. Partition Tolerance is the
application's ability to continue to function
in the event a network partition occurs.
BITS Pilani, Pilani Campus
CAP Theorem
A distributed system always needs to be partition tolerant, we shouldn’t be making a
system where a network partition brings down the whole system.
So, a distributed system is always built Partition Tolerant.
So, In simple words, CAP theorem means if there is network partition and if you want
your system to keep functioning you can provide either Availability or Consistency and
not both.
https://bikas-katwal.medium.com/mongodb-vs-cassandra-vs-rdbms-where-do-they-stand-in-the-cap-theorem-1bae779a7a15
BITS Pilani, Pilani Campus
Dealing with Network Partitions
Sacrifice Consistency-AP(maintain Availability and Partition tolerance). In this
scenario we allow writes on both sides of the partition. When the partition has been
resolved, both sides of the partition need to merge their data to return to a
consistent state.
Sacrifice Availability-CP(maintain Consistency and Partition tolerance)In this
scenario, after a partition has occurred, one side of the partition is disabled to
prevent inconsistency. The remaining partition remains consistent, and the
application continues with a lower degree of availability.
CA is not possible in distributed systems-it requires a single process
BITS Pilani, Pilani Campus
Eventual vs Strong Consistency
Strong Consistency - simply means that all the nodes across the world should
contain the same value for an entity at any point in time.
Eventual Consistency - at some point the system will let all users read the most
recently made updates, but all will not do so immediately. This greatly reduces
synchronism in our application and allows the system to remain available to all
users, even in the face of network partitions - during the partition, users may see
inconsistent data, but as the partition heals the consistency of the system will
eventually be restored. [Decide level of staleness that is acceptable]
BITS Pilani, Pilani Campus
Guidelines for building highly
scalable systems
1. Avoid shared resources as they might become bottleneck
2. Avoid slow services
3. Scaling Data tier is tricky
4. Cache is the key
5. Monitoring is important
BITS Pilani, Pilani Campus
Architecture’s Scalability
Requirements
Identify Scalability requirements early in the software life cycle so that it allows the
architectural framework to become sound as the development proceeds.
System scalability criteria could include the ability to accommodate:
• Increasing number of users [Ex: Facebook]
• Increasing number of transactions per millisecond [Ex: Banking applications]
• Increase in the amount of data [Ex: Instagram Reels / Amazon Prime / Flipkart]
BITS Pilani, Pilani Campus
Challenges for Scalability
1. Centralized Approach
2. Synchronous Communication
3. Cost
BITS Pilani, Pilani Campus
Case Study 1 - Uber
Uber Monolithic Architecture (Previous)
• A REST API is present with which
the passenger and driver connect.
• Three different adapters are used
with API within them, to perform
actions such as billing, payments,
sending emails/messages that we
see when we book a cab.
• A MySQL database to store all
their data.
Source - https://dzone.com/articles/microservice-architecture-learn-build-and-deploy-a
BITS Pilani, Pilani Campus
Basic Microservices Architecture
Source - https://dzone.com/articles/microservice-architecture-learn-build-and-deploy-a
BITS Pilani, Pilani Campus
Case Study 1 - Uber
Challenges of Monolithic Application
1. All the features had to be re-built, deployed and tested again and again to
update a single feature.
2. Fixing bugs became extremely difficult in a single repository as developers had
to change the code again and again.
3. Scaling the features simultaneously with the introduction of new features
worldwide was quite tough to be handled together.
Source - https://dzone.com/articles/microservice-architecture-learn-build-and-deploy-a
BITS Pilani, Pilani Campus
Case Study 1 - Uber
Uber Microservices Architecture (Current)
BITS Pilani, Pilani Campus
Case Study 1 - Uber
Advantages of Microservices Architecture
1. The units are individual separate deployable units performing separate functionalities.
For Example: If you want to change anything in the billing microservices, then you just have to deploy only
billing microservices and don’t have to deploy the others.
2. All the features were now scaled individually i.e. The interdependency between each and every
feature was removed.
For Example, we all know that the number of people searching for cabs is more comparatively more than
the people actually booking a cab and making payments. This gets us an inference that the number of
processes working on the passenger management microservice is more than the number of processes
working on payments.
Source - https://dzone.com/articles/microservice-architecture-learn-build-and-deploy-a
BITS Pilani, Pilani Campus
Case Study 2 - Netflix
• Netflix is a subscription-based video-on-demand OTT streaming service
• Handles large content of movies, television content, web series
• Has 180 Million subscribers across 200+ countries
What is Netflix Architecture, and how does it provide scalability?
Source - https://www.geeksforgeeks.org/system-design-netflix-a-complete-architecture/
BITS Pilani, Pilani Campus
Case Study 2 - Netflix
BITS Pilani, Pilani Campus
Case Study 2 - Netflix
• Open Connect CDN (Content Delivery Network)
• AWS Cloud
• Client device – Laptop / Mobile phone
• Backend services (includes database)
Source - https://www.geeksforgeeks.org/system-design-netflix-a-complete-architecture/
BITS Pilani, Pilani Campus
Case Study 2 - Netflix
Video Encoding
BITS Pilani, Pilani Campus
Case Study 2 - Netflix
• AWS Services – Login, Recommendation, Search, User History, Home Page,
Billing, Customer Support
Other components
• ZUUL – Gateway Service that provides dynamic routing based on the input
request
• Inbound filter – Authentication, Routing
• Endpoint filter – Return static response or forward to backend service
• Outbound filter – Zip content and calculate metrics. Send response to Netty Web
server and to the client
BITS Pilani, Pilani Campus
Case Study 2 - Netflix
Other components
Microservices
• Critical services – searching video, navigating to video, play video
• Make it independent from other services
Database
• MYSQL (RDBMS) – Data for billing information, user information and transaction
information. Deployed on ec2 instances.
• Cassandra (NOSQL) – Viewing history data (over 50 Cassandra Clusters with
500 nodes)
BITS Pilani, Pilani Campus
References
1) Building Scalable Systems by Ian Gorton
2) The Art of Scalability: Scalable Web Architecture, Processes, and Organizations
for the Modern Enterprise, Second Edition by Michael T. Fisher; Martin L. Abbott
Published by Addison-Wesley Professional, 2015
3) Microservices patterns by Chris Richardson, Manning Publications 2018
BITS Pilani, Pilani Campus
Summary
1. What is Scalability
2. Need for Scalable Architectures
3. Principles of Scalability
4. Scale Cube
5. CAP Theorem
6. Guidelines for Building Highly Scalable Systems
7. Architecture’s Scalability Requirements
8. Challenges for Scalability
9. Case Study – Uber / Netflix
BITS Pilani, Pilani Campus
Thank You!
