A New Hope

Monitoring
data.fao.org

http://www.ironicsans.com/images/anewhope.png

Ohai!

I am Bryan W. Berry

• I am in charge of infrastructure operations for http://data.fao.org, you probably already knew this

What does that mean?

• Provision new servers
• Automate everything possible
• Handle backups
• Also, responsible for monitoring

You can reach me at bryan.berry@fao.org or @bryanwb

Overview

• What we have now
• Why it sucks
• What might be better
• What you can do

We have a complicated application stack

systematic monitoring is critical to untangling it

http://dummyatcooking.files.wordpress.com/2007/10/spaghetti-bolognese.jpg

We currently use Nagios and RHQ

Downsides

• Graphing capabilities in Nagios+Cacti very limited
• Nagios is a pain to configure, even with Chef
• RHQ is powerful but very inflexible

More RHQ Issues

• The RHQ agent consumes non-trivial amount of RAM and CPU
• Extending it requires you to write Java class and a Maven pom.xml
• Small user community despite being a relatively mature project
• No easy way to access data in RHQ to create dashboards
  or do computations on data

That Makes me Want to...

http://3.bp.blogspot.com/-RAPTS9vZKKw/TZB8Zf7HSpI/AAAAAAAAEmA/5uSAjc4xfsc/s1600/the-scream.jpg

A New Hope

• A more flexible monitoring system
• We will continue to use Nagios for alerting
• The future use of RHQ is uncertain

The DevOps Community has (mostly) converged on a toolchain for monitoring

What is a metric?

• a name
• a value
• a timestamp, typically the UNIX epoch time

stats.haproxy.data_fao_org.request_duration 330 74857843

<stat_name> <number> <timestamp>

Collectd for system metrics

Collectd is good

• lightweight C daemon
• monitors continuously, not once every 5 minutes
• Best for OS-level metrics such as CPU, Disk, Memory, etc.

What's Graphite?

• Time Series Database (Whisper)
• Rendering Engine
• Dashboard (Graphite-Web)
• data relay and aggregation (Carbon)

Here is a taste

We'll come back to this later!

Let's take a look at that JVM

JMXTrans

• Is just a connector for transporting JMX data
• No agent involved

What about those logs?

We don't only care about metrics, we also care about important events

Scraping metrics from logs would be nice

Show me the graphic already!

Logstash can more than just ship logs

• Win

  • index by field
  • shape data
  • add new fields and tags to entries
  • Elasticsearch backend is awwwes0me

• Con - The agent is not light on resource usage

Logs don't have to be Ugly

We can filter the data

UNIX Tail in your browser

Elasticsearch Rocks

• We can use Lucene Parser Syntax to construct queries
• Watch out though, don't use quotes, the 1st example here works, the second doesn't

But this is not really enough

Ideally, your application should be instrumented from the inside

No one knows your code better than you

Use the Force, Luke

Use Statsd

http://www.alexandgregory.com/images/luke%20skywalker.jpg

Show me a graphic

Enter Statsd

• a small local daemon that your application send metrics to over UDP
• has virtually no overhead
• I use Pete Fritchman's Ruby implementation
• Statsd ships the metrics it receives to graphite
• Types of metrics
  • gauges
  • timers
  • counters

So about the code

Here is some sample java code

Metrics, Metrics Everywhere

Coda Hale gave an excellent talk about how his team at yammer uses metrics

He also created an excellent java library that you can use together with statsd

Now, Graphite Demo

show how to display requestduration across multiple subdomains

You will get for free

• System-level monitoring with collectd
• JMX monitoring w/ JMXtrans
• log aggregation on request via logstash+elasticsearch+kibana
• All those data points in graphite

But if you are serious about performance . . .

. . . You will

• instrument your code with the statsd java client or Coda Hale's metrics library
• Create custom graphs in graphite
• Solve performance problems the way real engineers do, with data

Questions?

Ask away

The Full Stack

Further Resources

• Logstash
• Collectd
• statsd and Pete Fritchman's ruby-statsd
• graphite
• Coda Hale's excellent talk and library java library
• Be sure to listen to the FoodFightShow!
  • Monitoring for n00bs
  • Monitoringsucks

Even Further Resources

All the cookbooks used

• chef-logstash
• cookbook-elasticsearch
• chef-collectd
• chef-jmxtrans
• chef-ruby-statsd
• chef-graphite