An Architect's Guide to Site

Reliability Engineering
Nathaniel T. Schutta
@ntschutta
ntschutta.io
Software development practices evolve.
Feature not a bug.
It is the agile thing to do!
We’ve gone from devs and ops
separated by a large wall…
To DevOps all the things.
 We’ve gone from monoliths to
service oriented to microserivces.
And it isn’t all puppies and rainbows.
Shoot.
A new role is emerging - the
site reliability engineer.
Why?
What does that mean to our teams?
 What principles and
practices should we adopt?
How do we work together?
What is SRE?
Important to understand the history.
Not a new concept but a catchy name!
Arguably goes back to
the Apollo program.
Margaret Hamilton.
Crashed a simulator by inadvertently
running a prelaunch program.
That wipes out the navigation data.
Recalculating…
Hamilton wanted to add error-checking
code to the Apollo system that would
prevent this from messing up the systems.
But that seemed excessive to her higher-
ups. “Everyone said, ‘That would never
happen,’” Hamilton remembers.
But it did. Right around Christmas 1968.
— ROBERT MCMILLAN

https://www.wired.com/2015/10/margaret-hamilton-nasa-apollo/
 Luckily she did manage to
update the documentation.
Allowed them to recover the data.
Doubt that would have turned into a
Hollywood blockbuster…
Hope is not a strategy.
But it is what rebellions are built on.
Failures, uh find a way.
 Traditionally, systems
were run by sys admins.
AKA Prod Ops. Or something similar.
And that worked OK. For a while.
But look around your world today.
Service all the things!
Seems like everything is
aaS these days…
Infrastructure. Container. Platform.
Software. Function. Pizza.
Pretty sure about that last one…
Architecture as a Service…
Nothing new really.
CORBA anyone?
Facilitate communication for disparate
systems on diverse platforms…
Too soon?
EJB then.
Still have flashbacks?
Sorry. I’ve tried to forget most of it.
It didn’t stop there though did it?
Remember when SOA was
the new hotness?
Everything had to be all
SOA all the time!
There were books and talks and blogs
(remember those?), it was great!
How about API first?
Popularized in certain quarters.
Helpful?
How about this one?
Bet you use that one everyday.
Maybe without knowing it.
One of these perhaps?
What caused this Cambrian
explosion of APIs?
Technology changes.
Proliferation of computers in
everyone’s pockets.
Commoditized hardware.
The Cloud!
Companies were altering
their approach too.
Little company called Amazon
made a major change.
Well, it had happened years earlier.
But a publicly shared rant detailed it.

https://plus.google.com/+RipRowan/posts/eVeouesvaVX
Steve Yegge - the Bezos mandate.
All data will be exposed through a
public service interface.
 These interfaces are *the*
communication method between teams.
No other form of communication is
allowed. No direct reads, links etc.
No back doors.
All services must be designed to
be public. No exceptions.
Don’t want to do this? You’re fired.
Unsurprisingly, things began to change.
And we learned some things.
Calls bounce between 20 services…
where was the problem?
Who do we page?
How do we monitor this
web of services?
How do I even *find*
these other services?
Debugging gets harder…
We’ve seen this story continue today.
Can’t swing a dry erase marker without
hitting someone talking about…
Microservices!
Bounded Context.
Domain-Driven Design.
 Arguments over the
definition of microservice…
https://mobile.twitter.com/littleidea/status/500005289241108480
Rewrite it in two weeks.
Miniservice. Picoservice.
What do we even mean by
“application” today?!?
How about functions then?
Have we found a golden hammer yet?
Bet that would be helpful during
your next retro!
Turns out there are still engineering
issues we have to overcome.
It isn’t all puppies and rainbows.
Sorry.
Turns out, those things Yegge
mentions…are still things.
 What would you say your
microservice call pattern looks like?
http://evolutionaryarchitecture.com
The traditional sys admin approach
doesn’t give us reliable services.
Inherent tension.
Conflicting incentives.
 Developers want to
release early, release often.
Always Be Changing.
But sys admins want stability.
It works. No one touch anything.
Thus trench warfare.
Doesn’t have to be this way!
We can all get along.
What if we took a different
approach to operations?
“what happens when you ask a software
engineer to design an operations team.”

https://landing.google.com/sre/book/chapters/introduction.html
 Ultimately, this is just software
engineering applied to operations.
Replace manual tasks with automation.
Focus on engineering.
Many SREs are software engineers.
 Helps to understand UNIX
internals or the networking stack.
Our operational approach has to evolve.
The “Review Board” meeting
once a quarter won’t cut it.
How do we move fast safely?
 Operations must be able to
support a dynamic environment.
That is the core of what we mean by
site reliability engineering.
How we create a stable, reliable
environment for our services.
It doesn’t happen in spare cycles.
Make sure your SREs have time to
do actual engineering work.
 On call, tickets, manual tasks -
shouldn’t eat up 100% of their day.
SREs need to focus on automating
away “toil” aka manual work.
Isn’t this just DevOps?
 Can argue it is a natural
extension of the concept.
 Think of SRE as a specific
implementation of DevOps.
SRE Responsibilities.
What should my SRE team focus on?
Availability. Stability. Latency.
Performance. Monitoring.
 Capacity planning.
Emergency response.
Drive automation.
SREs help us establish our SLOs.
Embrace risk. Manage risk.
Risk is a continuum.
And a business decision.
What do our customers expect?
What do our competitors provide?
Cost.
Should those sites/apps have
had a redundant backup?
https://twitter.com/KentBeck/status/596007846887628801
How much would that have
cost those sites/apps?
How much more revenue would
that have driven for them?
It is a tradeoff.
Long term vs. short term thinking.
Heroic efforts can work
for the short term.
But that isn’t sustainable.
 In the long run it may be better to
lower your SLOs for a short time…
To allow you to engineer a
better long term solution.
Mean time to recovery.
Having a run book helps immensely.
…thinking through and recording the
best practices ahead of time in a
"playbook" produces roughly a 3x
improvement in MTTR as compared to
the strategy of "winging it."

— Benjamin Treynor Sloss

Site Reliability Engineering
We don't rise to the level of our
expectations, we fall to the level of our
training.

— Archilochus
https://mobile.twitter.com/walfieee/status/953848431184875520
 Design/implement
monitoring solutions.
Establish alerting.
Logging best practices.
Create dashboards.
 Four Golden Signals.

https://landing.google.com/sre/book/chapters/monitoring-distributed-
systems.html#xref_monitoring_golden-signals
Latency - how long does it take to
service a request.
 Traffic - level of demand on the
system. Requests/second. I/O rate.
 Errors - failed requests. Can be
explicit, implicit or policy failure.
 Saturation - how much of a
constrained resource do we have left.
Important to consider the
sampling frequency.
High resolution can be costly.
Aggregate data.
Some measures benefit from shorter
intervals…others not so much.
Establish alerting thresholds.
Alert levels can be hard to get right.
Should be actionable.
Should require a human.
Should result in a sense of urgency.
Implies we cannot have more than a
few pages a day - people burn out.
You can over alert and over monitor!
Eliminate toil.
Automate all the things.
Manual. Repetitive. Automatable.
One offs. Reactive. Grunt work.
Toil drives people out of SRE work.
Hurts morale. Sets a bad precedent.
People can’t do the same thing twice.
See golf.
We need consistency.
Deployment pipeline has
to be repeatable.
We cannot move fast safely unless
SREs are freed from toil.
Postmortems.
We will make mistakes.
Outages will still happen.
Vital we learn from those experiences.
Do not blamestorm.
“Blameless postmortems.”
Goal is to prevent it from
happening again.
Document the incident.
What happened?
What was the root cause(s)?
What can we do to prevent this
from happening in the future?
Be constructive, not sarcastic.
Consider a basic template.
Title/ID.
Authors.
Status.
Impact.
Root Causes.
Resolution.
Action Items.
Lessons Learned.
Timeline.
Whatever you think will help!
Can be difficult to create a
postmortem culture.
Consider a postmortem of the month.
Book club.
Wheel of Misfortune.
Role play a disaster you faced before.
Ease into it.
Recognize people for their participation.
Senior management needs to
encourage the behavior!
Perform retros on your postmortems!
Improve them!
We cannot learn anything without first
not knowing something.

— Mark Manson
The Subtle Art of Not Giving a F*ck
All services are equal. Some services
are more equal than others.
 Defining our SLOs is a critical step
towards production hardened services.
Availability is one of our most
important objectives.
What is the availability goal
for this specific service?
SLAs vs. SLOs.
Terminology is often misused.
SLI - service level indicator.
 A measure of some
aspect of our system.
Latency, error rate, throughput…
Availability.
Percentage of time your
service is available.
How much downtime can you tolerate?
99%: 7.20 hours a month,
14.4 minutes a day.
99.9%: 8.76 hours PER YEAR.
1.44 minutes a day.
99.99%: 4.38 minutes A MONTH.
8.66 seconds a day.
99.999%: 6.05 seconds A WEEK,
864.3 milliseconds a day.
By the way, the GKE’s availability
SLA is 99.5%. Just saying.

https://cloud.google.com/kubernetes-engine/sla
SLO - service level objective.
Our target value or a range of values.
Our SLO for availability is 99.9%.
Our average response time should be
less than 100 milliseconds.
Can be very tricky to pick an SLO!
More on that in a minute.
We don’t always get to choose our SLOs.
Our users may have a thing or two
to say about what they want!
We are also subject to the
things we depend on.
If you depend on a service that only
guarantees 99.9% availability…
You cannot guarantee 99.99%!
SLA - service level agreement.
Assign a consequence to
missing/meeting an SLO.
Often contractual and involve
some kind of financial penalty.
If there is no consequence,
we’re talking about an SLO.
 SLAs are the realm of
product and legal decisions.
Many people say SLA
when they mean SLO.
Pedantic much?
Ubiquitous language and all that…
SLOs are not a purely technical decision.
Our business and our customers will
have an awful lot to say here.
Simple is good.
Infinity is not a goal.
More is not better.
SLOs should help you prioritize.
 Give yourself some wiggle room -
you can always tighten them up later.
Ensure there is some margin for error.
May have an “internal” SLO that is
tighter than your advertised SLO.
Overachieving doesn’t help.
People will start to rely on
your overachieving!
 Introduce some latency or
downtime to stay at your SLO.
Everyone wants 99.999%.
Everyone wants hot/hot.
Until they see the price tag.
If you have to ask…
Establish an error budget.
Establish your target availability.
Say 99%.
Your error budget is 1.0%.
Spend it however you want! Just
don’t go over that limit.
Goal is not zero outages. Goal is to
stay within the error budget.
Go ahead with those experiments.
Try different things.
Once we use up the budget though…
Have to slow our roll.
Aligns incentives.
Helps everyone understand
the cost/benefit tradeoff.
Working with SREs.
Production Readiness Reviews.
Not a one time, up front thing.
Services should be reviewed
and audited regularly.
Does not have to be high ceremony!
Get the team together - SREs, Devs,
etc. Draw up the architecture.
Do we have a shared understanding
of what the system does?
Do we have a shared understanding
of our requirements?
As we talk through it, we
will discover bottlenecks.
The Wombat service has a lower
availability level than we need.
We will find interesting failure cases.
“When month end falls on the
Super Blue Blood Moon.”
Review should result in a new
architecture diagram or two.
And probably some new
items on the backlog.
When we refer to an application or
microservice as “production-ready,” we
confer a great deal of trust upon it: we
trust it to behave reasonably, we trust it
to perform reliably, we trust it to get the
job done…
— Susan J. Fowler
Production-Ready Microservices
How do we know we
can trust a service?
Consider having a checklist.
A checklist? Seriously?
http://atulgawande.com/book/the-checklist-manifesto/
You know who uses checklists?
Pilots. Surgeons.
Should be quantifiable and measurable.
“Fast” won’t cut it.
Stability.
Reliability.
Scalability.
Fault tolerance.
Performance.
Monitoring.
Documentation.
I know what some of you are thinking…
I don’t have time for all this.
We need to MOVE FAST.
And break things…
We’re Agile. With a capital A.
How is your velocity when an outage
brings your business to a halt?
Requires buy in from the grass roots
level as well as management.
Perform an audit.
Go back to your checklist. Does the
service meet our requirements?
 Probably results in some
new things in our backlog!
 Now we can create a
production readiness roadmap.
What do we need to fix and
when can/should we fix it.
Drive prioritization of the work.
 A lot of this is manual. But
some of it can be automated!
http://evolutionaryarchitecture.com
Fitness functions!
Basically, a set of tests we execute
to validate our architecture.
 How close does this particular
design get us to our objectives?
Ideally, all automated. But we may
need some manual verifications.
For example…
 All service calls must
respond within 100 ms.
Cyclomatic complexity
shall not exceed X.
Hard failure of an application
will spin up a new instance.
Reviews and audits should
not be additional red tape.
Should not be overly bureaucratic.
Couple of hours…depending on
the complexity of the system.
Architectural reviews.
Look for failure points.
Draw up the architecture.
What happens if *this* fails?
It can’t fail? Yeah it can -
what happens if it does?
Think through how our
service could fail.
 It is hard. We are really good at
thinking through the happy path.
But we need to think about
the road less traveled.
Test for these failure scenarios. Does
our service respond appropriately?
Only one way to really know…
https://github.com/Netflix/SimianArmy
Chaos engineering.

http://principlesofchaos.org
Next steps.
Do you have an SRE team?
Do you wish you had an SRE team?
It can be built!
You probably have some engineers
that can (and would!) do it.
Our applications are changing radically.
Dev Ops
We all need to evolve to succeed.
We can’t move fast safely unless the
environment enables it.
We have to work together.
Want more?
https://landing.google.com/sre/book.html
Questions?
 Thanks!
I’m a Software Presentation Modeling for
Architect, Patterns Software
Now What?
with Nate Shutta
with Neal Ford & Nate Schutta
Architects
with Nate Shutta Nathaniel T. Schutta
@ntschutta
ntschutta.io