Metal Toad Managed Services Dashboard - Part 2: Tools for Logging

In Part 1 we saw that the layout showed a glimpse of what is behind the scenes of our new managed services dashboard, the graphs were all generated with the help of a utility called Graphite. Graphite can't do the job on its own though, it takes a whole host of other programs to get the data from the various servers to Graphite in a form that it can read.

1. Logstash: The first program in the chain is Logstash. Logstash is the program responsible for reading data from the various logs, changing the formatting, and then sending it to our log aggregation server. The best part is that logstash is almost endlessly versatile. Logstash can read data in several different formats, manipulate it, and then output data in other formats. Currently, we read custom logs for system stats like server load, memory, and Apache logs in json format. Logstash receives this data and sends it to the log aggregation server to be temporarily stored in Redis.
2. Redis: Redis in itself, is a powerful tool used for storing key-value pairs. We use Redis to take the data stream from multiple sources and combine them into a single data stream.
3. Logstash: After Redis has created the single data stream, Logstash sees the data a second time, this time it only deals with data on the log aggregation server, it takes output from Redis and outputs it to StatsD and Elasticsearch.
4. Statsd: StatsD, a program developed by Etsy, is the data engine that allows us to send data to Graphite.
5. Graphite: Finally, we end with Graphite. Graphite allows us to give a visual representation to our single stream data from StasD. With a few clicks of the mouse, you can perform mathematic formulas on the data. For example, the dashboard graphs bandwidth from a single server, the total errors from all servers, and the average resource usage across multiple servers.