Metal Toad Managed Services Dashboard - Part 2: Tools for Logging

In Part 1 we saw that the layout showed a glimpse of what is behind the scenes of our new dashboard, the graphs were all generated with the help of a utility called Graphite. Graphite can't do the job on its own though, it takes a whole host of other programs to get the data from the various servers to Graphite in a form that it can read.

  1. Logstash: The first program in the chain is Logstash. Logstash is the program responsible for reading data from the various logs, changing the formatting, and then sending it to our log aggregation server. The best part is that logstash is almost endlessly versatile. Logstash can read data in several different formats, manipulate it, and then output data in other formats. Currently, we read custom logs for system stats like server load, memory, and Apache logs in json format. Logstash receives this data and sends it to the log aggregation server to be temporarily stored in Redis.
  2. Redis: Redis in itself, is a powerful tool used for storing key-value pairs. We use Redis to take the data stream from multiple sources and combine them into a single data stream.
  3. Logstash: After Redis has created the single data stream, Logstash sees the data a second time, this time it only deals with data on the log aggregation server, it takes output from Redis and outputs it to StatsD and Elasticsearch.
  4. Statsd: StatsD, a program developed by Etsy, is the data engine that allows us to send data to Graphite.
  5. Graphite: Finally, we end with Graphite. Graphite allows us to give a visual representation to our single stream data from StasD. With a few clicks of the mouse, you can perform mathematic formulas on the data. For example, the dashboard graphs bandwidth from a single server, the total errors from all servers, and the average resource usage across multiple servers.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.

About the Author

Nathan Wilkerson, Senior Cloud Engineer

Nathan started building computers, programming and networking with a home IPX network at age 13. Since then he has had a love of all things computer, taking classes programming before transitioning to networking and system administration.

When not at work Nathan can be found chasing after his two kids or reading a book.