Measure it,
Don't Just Monitor It alt text

Devopsdays Italy 2012, Bryan W. Berry

Ohai There

I work at UN Food and Agriculture Organization on the http://data.fao.org project
Creator and co-host of the Food Fight podcast
You can reach me at bryan.berry@gmail.com or @bryanwb
I am a novice at statistics, graphs, and monitoring

Overview

SOA is complicated
Monitoring Helps
But What does it all mean?
Measure it, don't just monitor it
Presentation matters

data.fao.org Architecture on Paper

The Reality

http://dummyatcooking.files.wordpress.com/2007/10/spaghetti-bolognese.jpg

What We use for Monitoring

Collectd (cpu, disk, network, arbitrary system data)
JMXtrans to poll JMX data
Logstash + statsd awesome combination
Graphite
Elasticsearch + Kibana

largely based on discussion in foodfightshow episode 21

Here is what it looks like

Let There Be Graphs!

Do you know What this means?

Just collecting the data isn't enough

How fast/slow is the site?

How many Requests per Second? Per minute?
Are those thousands of seconds or thousands of milliseconds?
What is the average request latency?
Is "average" what we really want to know?

http://www.buzzlol.com/wp-content/uploads/2012/05/Confused-Dog-Is-Confused.jpg

Back to Basics, what is a metric?

a bucket
a value
a timestamp, typically the UNIX epoch time

bucket.name number TIMESTAMP

<stat_name> <number> <timestamp>

Metric Types for Collectd

Gauges
Derive
Counters
Absolute

Gauges are the most commonly used metric type

counters

Metric Types for Statsd

Gauges
Counters*
Timers

Counters* different than counters in collectd

counters

Really Basic

basic transport

collectd in action

collectd

statsd in action

Gauges

A gauge simply indicates
an arbitrary value at a point
in time and is the
simplest type in StatsD and collectd

http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/

Derive metric type (collectd only)

Computes the delta of an underlying counter for a period of time
Very useful for measuring network traffic

value = \frac{{value}_{new} - {value}_{old}}{{time}_{new} - {time}_{old}}

Counters (statsd)

A counter adds a value to a bucket and stays in memory until the flush interval
it sends two numbers for each bucket, a total count for the flush interval and a value per second

counter

http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/

Statsd Timers

Collect series of values
- raw count
- lower - minimum value for flush interval
- mean
- upper - maximum value for flush interval
- 90th percentile, value at 90th percentile

timer

Let's take a look that JVM

jmxtrans

JMXtrans

JMXTrans polls each JVM each 60 seconds, by default
As far as I can tell, values returned from a JMX query are Gauges

Let's look at that graph again

That's horrible!

Averages are Nasty, MEAN things

Let's look at that 90th Percentile

Let's remove the 90th Percentile

learn it Love it

removeAbovePercentile, along with other filtering functions, is freaking awesome
other filtering functions
- removeAboveValue
- removeBelowPercentile
- removeBelowValue
- mostDeviant

The average isn't all it's cracked up to be

These functions are broken in Graphite 0.9.10 but fixed in master

Show me the worst!

Measure it

Know your data
Filter out distorting data
Smooth the Savage Graph to reveal trends
Maximize the power of Graphite

I see spikes, but no trend

spikes

Smoothing

Massage the Numbers to reveal the trend and to make useful forecasts

Moving Average
Moving Weighted Averages
Holt-Winters

Let's Move that Average

moving average

erm, a little better

Show me the Math!

For value M at time t and N number of points, calculate the average based on the last 10 points

M_{t} =
[X_{t} + X_{t-1} + ... + X_{t-N+1}] / N

Moving Averages are OK at reflecting trend,
but have no notion of seasonality

Got Seasonality?

more hw

http://blog.pkhamre.com/2012/07/05/visualizing-logdata-with-logstash-statsd-and-graphite/

Holt-Winters forecast

Let's Factor in:

How smooth do we want the data to be, i.e. not spiky
Trend
Seasonality

Now with Seasoning!

more hw
Graph credit: Pal Kristian Hamre

http://blog.pkhamre.com/2012/07/05/visualizing-logdata-with-logstash-statsd-and-graphite/

Show me the Math!

Idiot's guide to Holt Winters

4 Equations
- Overrall Smoothing
- Trend Smoothing
- Seasonal Smoothing
- Actual Forecast
3 Constants
- α - overrall
- β - Trend
- γ - Seasonality

http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc435.htm

Confidence Bands

confidence bands
Graph credit: R.I. Pienaar

https://github.com/ripienaar/graphite-graph-dsl/wiki/Creating-Holt-Winters-Forecasts

Aberration

aberration
Graph credit: R.I. Pienaar

Great potential in Holt-Winters

Intelligent capacity planning!
More intelligent alerting
Problems
- Buggy implementation
- Need to null outage data

Deming Matters

Deming invented the Control Chart
100 years ago

Presentation matters

Maximize Data-ink Ratio
Maximize data density
Use words and colors (to a lesser extent) to add context to your graph

Edward Tufte, Data Visualization Guru

"Graphics should do more than present the obvious to idiots"

More wisdom

"The task of the designer is to give visual access to the subtle and the difficult -- that is, the revelation of the complex"

Maximize Data-ink Ratio

Drop Unnecessary Grid lines

Ah, much better!

Colors and Text can Illuminate

More Interesting Graphite Functions

scale
removeAbovePercentile
cumulative
summarize
hitCount
. . . and many more!!

John Rauser - look at your data

Keep your logs, ideally in elasticsearch

The Power of Chef

I couldn't have put together this stack w/out Chef
cookbooks online and available
Checkout monigusto!

Further Resources

Graphite Functions
Holt-Winters Approach to Exponential Smoothing by F. Dekhordi
The Visual Display of Quantitative Information by Edward Tufte
Pal Kristian Hamre's blog
Jason Dixon of course
Listen to the FoodFightShow Podcast
- Monitoring for n00bs
- Monitoringsucks

Special Thanks to

Jason Dixon
Chris Davis
Pal Kristian Hamre
Pete Fritchman
Michael Leinartas

This presentation is available on github in showoff format (html,css,etc.), and diagrams done in SVG

z, ?	toggle help (this)
space, →	next slide
shift-space, ←	previous slide
d	toggle debug mode
## <ret>	go to slide #
c, t	table of contents (vi)
f	toggle footer
r	reload slides
n	toggle notes
p	run preshow
P	toggle pause

Ohai There

more on data.fao.org

Overview

data.fao.org Architecture on Paper

What We use for Monitoring

Here is what it looks like

Let There Be Graphs!

Do you know What this means?

How fast/slow is the site?

Back to Basics, what is a metric?

Metric Types for Collectd

Metric Types for Statsd

Really Basic

collectd in action

statsd in action

Gauges

Derive metric type (collectd only)

Counters (statsd)

Statsd Timers

Let's take a look that JVM

JMXtrans

Let's look at that graph again

Averages are Nasty, MEAN things

Let's remove the 90th Percentile

learn it Love it

Show me the worst!

Measure it

I see spikes, but no trend

Smoothing

Let's Move that Average

Show me the Math!

Got Seasonality?

Holt-Winters forecast

Now with Seasoning!

Show me the Math!

Idiot's guide to Holt Winters

Confidence Bands

Aberration

Great potential in Holt-Winters

Presentation matters

Edward Tufte, Data Visualization Guru

More wisdom

Maximize Data-ink Ratio

Colors and Text can Illuminate

More Interesting Graphite Functions

John Rauser - look at your data

The Power of Chef

Further Resources

Special Thanks to